MAGMA: Manifold Regularization for MAEs

Alin Dondera; Anuj Singh; Hadi Jamali-Rad

REVIEW 5 major objections 5 minor 39 references

MAGMA: Manifold Regularization for MAEs

T0 review · 5 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper introduces MAGMA, a batch-wide layer-wise manifold-regularization loss added during masked-autoencoder pretraining, and claims it raises linear-probing accuracy by up to 11 points on ImageNet-100 and gives consistent gains…

desk verdict A clean, cheap regularizer for MAE that shows promising gains, but the headline numbers rest on favorable test-set hyperparameter tuning and no error bars. read the letter →

arxiv 2412.02871 v2 pith:VHYK4YM3 submitted 2024-12-03 cs.CV

Alin Dondera , Anuj Singh , Hadi Jamali-Rad This is my paper · ORCID

classification cs.CV

keywords maskedautoencodersself-supervisedlearningmanifoldregularizationvisiontransformerslinearprobinglayer-wiserepresentationcontrastive

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes a simple modification to masked-autoencoder (MAE) pretraining: a batch-wide, layer-wise manifold-regularization loss that encourages representations in one Transformer layer to preserve the pairwise geometry of representations in another layer. The authors argue that MAE features are unregularized compared with CNN features, and that this added term fills that gap. They report consistent gains in linear probing and k-nearest-neighbour accuracy across CIFAR-100, STL-10, Tiny-ImageNet, and ImageNet-100, plus larger gains in low-data regimes. The same loss also gives smaller, mostly marginal improvements when added to VICReg and SimCLR. If the claim holds, a drop-in loss during pretraining is enough to improve MAE-based representations without changing the architecture.

What carries the argument

The central object is a batch-wide layer-wise manifold-regularization loss, defined on a pair of layers $(k,l)$ as $\mathcal{L}_{\mathrm{Reg}} = \frac{1}{B^2}\sum_{i,j} w(Z_i^{(k)}, Z_j^{(k)})\, \lVert Z_i^{(l)} - Z_j^{(l)}\rVert^2$, where $w$ is the RBF kernel over pairwise distances in the reference layer $k$ and $Z_i^{(l)}$ is the averaged patch representation of image $i$ at layer $l$. This is equivalent to a normalized Laplacian trace term $\frac{1}{B^2}\mathrm{Tr}(Z^{(l)T} L Z^{(l)})$. The loss penalizes pairs that are close in layer $k$ but far in layer $l$, effectively pulling representations in $l$ into alignment with the local geometry of $k$. The paper applies it with $k$ and $l$ as the penultimate and last layers of a ViT, using a dynamic kernel width $\sigma^2 = \mathrm{var}(\lVert Z_i^{(k)}-Z_j^{(k)}\rVert^2)$, and only during pretraining.

What would settle it

A reader can settle the claim by fixing the hyperparameters ($\lambda=1$, $e_{\mathrm{est}}=10$, $e_{\mathrm{dur}}=100$) and running MAE vs M-MAE on a dataset not used in the paper, such as Flowers-102; if M-MAE does not beat MAE by a substantial margin, the claimed general improvement is falsified.

Watch

Extended reading notes

Core claim

The central claim is that a single extra loss term, computed from the representations of two Transformer layers within each batch, can significantly improve the downstream classification performance of MAE-based models. Concretely, the paper reports that adding MAGMA to MAE raises linear probing accuracy by 5.1 points on CIFAR-100, 4.5 points on STL-10, 2.9 points on Tiny-ImageNet, and 11.0 points on ImageNet-100, with parallel gains for kNN accuracy. The regularization also improves U-MAE, a stronger MAE baseline, and yields smaller gains for SimCLR and VICReg. The authors attribute the effect to a reduction in representational noise and to more semantically coherent attention maps, and they show that the loss can be removed partway through pretraining with a persistent benefit.

Load-bearing premise

The reported improvements depend on the loss strength, start time, duration, and layer pair being tuned for each dataset without holding out a validation set; if those settings were tuned to the test sets instead, the gains could shrink or vanish on new data.

Editorial extensions

If this is right

Linear probing accuracy of MAE improves on all four evaluated datasets, by between 2.9 and 11.0 points, with similar kNN gains.
Low-data regimes benefit the most: on 1%, 5%, and 10% ImageNet subsets, M-MAE gains 1.1, 6.1, and 8.9 points over MAE respectively.
Ablations show the regularization weight $\lambda$ is the most sensitive hyperparameter; values above or below a narrow range erode the gain.
The loss adds only about 1.5–2% throughput cost and 100 MB GPU memory, with no change in parameter count.
The method is architecture-agnostic in principle and transfers to other SSL losses such as SimCLR and VICReg, though with smaller gains.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A testable implication the paper leaves implicit is that the gain should scale inversely with how much batch-wise information the pretext task already shares: contrastive methods, which already compare samples within a batch, should benefit less than MAE, as the paper's SimCLR/VICReg results hint.
The dynamic kernel width $\sigma^2 = \mathrm{var}(d_{ij})$ may be the key to training stability; a fixed width could either over-smooth or under-regularize, which could be tested by sweeping it independently.
Because the authors tune hyperparameters per dataset without a held-out split, a fairer test would fix $\lambda, e_{\mathrm{est}}, e_{\mathrm{dur}}$ across all datasets or select them on a validation set; readers should watch for such a re-evaluation.
A natural extension is to apply the same regularization to other generative or reconstruction-based SSL methods (e.g., BEiT or SimMIM), where the batch-wise geometry is similarly unused.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Colleague,

The quick take: MAGMA is a simple, nearly free layer-wise manifold-regularization loss that could be an easy win for MAE pretraining, but the headline gains are not yet trustworthy because the key weight λ was tuned on the test benchmarks themselves and there are no repeated seeds anywhere.

What's actually new: combining classical manifold regularization (Belkin et al.) with a cross-layer reference setup—regularize layer l using the RBF-similarity graph built on layer k—and applying that to MAE during pretraining. That specific construction is not in the cited literature. The paper does the right things around it: four datasets, linear and kNN evaluation, ablations on λ, warmup, duration, and layer pair, plus a limited but honest check on VICReg/SimCLR. The code is public, and the computational overhead is small (1.5–2% throughput drop). The ablations are genuinely informative.

The soft spots are real. No error bars or multiple seeds anywhere, which makes the word "significant" hard to parse. The stress-test point is accurate: on ImageNet-100, linear accuracy is 69.0 with λ=1, 60.0 with λ=0.1, and 54.6 with λ=0.01, against an MAE baseline of 58.0. So the advertised +11 points are almost entirely a result of picking λ=1 on the evaluation set. The supplementary says λ=1 is fixed for all datasets, but it was still selected by looking at the same test numbers. Without a validation split or seed variance, the large gains on smaller datasets and the 0.5-point gain on full ImageNet-1K could be partly tuning artifacts. The layer pair (penultimate as reference, last as regularized) also looks like it was chosen on these benchmarks.

I don't think this is fatal—the method is plausible, and even with λ=0.1 it beats the baseline by a couple points on ImageNet-100. But the paper oversells with "significantly," and the evidence as presented doesn't support the magnitude of the claimed improvements.

Who gets value: SSL practitioners looking for a cheap plug-in regularizer that might give a few points on small/medium datasets, and anyone interested in how much of empirical ML results is hyperparameter luck. It deserves a serious referee round, with the clear expectation that the authors add multiple seeds, report variance, and either fix λ a priori or select it on a proper validation split.

Recommendation: send it to peer review. The idea is worth testing, but the authors need those statistical anchors before the claims become credible.

Referee Report

5 major / 5 minor

Summary. This paper proposes MAGMA, a batch-wide layer-wise manifold regularization term added to the pretraining loss of masked autoencoders (and other SSL methods such as SimCLR and VICReg). The regularization uses an RBF kernel on representations of a reference layer to weight pairwise distances in a target layer, thereby penalizing pairs that are close in one layer but far in another. Experiments on CIFAR-100, STL-10, Tiny-ImageNet, and ImageNet-100 report gains in linear probing and kNN accuracy for MAE and U-MAE, with smaller gains for SimCLR and VICReg. The paper also ablates the regularization weight, the warmup period, and the duration of the regularizer, and provides qualitative visualizations (PaCMAP, PCA, attention maps).

Significance. If the reported gains are reproducible, MAGMA is a simple, low-cost addition to MAE pretraining that can improve learned representations without architectural changes or extra parameters. The computational overhead is small (about 1.5-2% throughput drop), and the idea of layer-wise manifold regularization is intuitive. However, the current evidence is weakened by the absence of error bars, the strong sensitivity of the results to the regularization weight, and the lack of a validation-based hyperparameter selection procedure. These issues need to be addressed before the central claim of consistent and significant improvement can be accepted.

major comments (5)

[Sec. 5.3 / Table 4] The headline improvement over MAE on ImageNet-100 (58.0 to 69.0, +11 points) is obtained only at lambda=1; at lambda=0.1 the accuracy is 60.0 (+2) and at lambda=0.01 it drops to 54.6, below the baseline. The supplementary material states that lambda is set to 1 for all MAE-based methods, but the ablation is run on the same ImageNet-100 evaluation set that produces the headline number, with no held-out validation split for hyperparameter selection. This makes the central claim vulnerable to selection bias; please provide results with a validation-based lambda choice or a fixed lambda across all datasets, together with the sensitivity plot.
[Tables 1-5] No error bars, standard deviations, or multiple runs are reported for any experiment. As many of the reported gains are small (e.g., Tiny-ImageNet linear +2.9 and kNN +2.8; SimCLR gains below 1 point on CIFAR-100 linear and kNN), the word 'significant' is not statistically justified. Report at least three independent seeds per configuration with mean and standard deviation, and run a paired significance test (e.g., Wilcoxon signed-rank) between the baseline and the regularized model.
[Table 3] On the full ImageNet-1K split (100% column), M-MAE outperforms MAE by only +0.5 (51.0 vs 50.5), which is an order of magnitude smaller than the ImageNet-100 gain and within the range of run-to-run noise given the absence of error bars. The baseline accuracy of 50.5 for MAE on ImageNet-1K linear probing is also unusually low, so the comparison may not reflect the standard MAE training setup. Please clarify the pretraining epochs and configuration for this experiment, report multiple seeds, and discuss the apparent lack of scaling benefit; qualify the central claim accordingly.
[Sec. 3.1, equations after Eq. (1)] The Laplacian reformulation is mathematically incorrect. For a symmetric weight matrix W, the identity is sum_{i,j} W_ij ||z_i - z_j||^2 = 2 Tr(Z^T (D - W) Z), with D the degree matrix. The paper instead defines L = D^{-1/2} W D^{-1/2} and writes L_Reg = (1/B^2) Tr(Z^T L Z); this is not the standard normalized Laplacian and does not reproduce the loss in Eq. (1). Either derive the correct relation or state plainly that Eq. (1) is computed directly and the Laplacian is used only as a conceptual tool.
[Sec. 5.2 / Fig. 3] The layer pair K is a crucial free parameter, but no quantitative ablation over layer pairs is provided. The claim that 'regularizing the last layer with respect to the penultimate layer seems to have the maximum impact' is unsupported by any table or figure with comparable numbers. Please include an ablation over reference and target layers, ideally on a validation set, and clarify whether the chosen pair is used across all datasets or tuned per dataset.

minor comments (5)

[Abstract and Introduction] 'Important divide' should be 'important branch' or 'important direction'; the paper also contains 'di-vide' hyphenation errors in the abstract and introduction.
[Fig. 3 caption] 'Last year' should be 'last layer'; the caption ('Effect of regularization. Implication: if the representations from any two layers are close, then the output representation will also be close.') does not describe what is plotted, so readers cannot interpret the figure; please add axes and a proper explanation.
[Sec. 3.1] The notation K, L, k, l is confusing: L denotes the number of layers, but l is also used as a layer index and as the regularized layer in K = {l,L}. Please rename the last layer index (e.g., L_last) and define K as a set of ordered layer pairs (k_ref, l_reg) to avoid the ambiguity in Eq. (1).
[Sec. 5.4] 'PacMAP' should be 'PaCMAP' (the method name is PaCMAP).
[Table 2] The fine-tuning results are not consistently improved; on CIFAR-100, M-MAE (75.6) is lower than MAE (76.9). The text says 'similar results,' which is acceptable, but the abstract's claim of improving 'performance' should be qualified to linear probing and kNN unless fine-tuning gains are also targeted.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: MAGMA's regularizer is an independent additive loss evaluated by external linear probing; hyperparameter tuning on the test benchmark is a robustness concern, not a circular derivation.

full rationale

The paper's central claim is an empirical one: adding the layer-wise manifold regularization loss L_Reg (Eq. 1) to MAE pretraining improves downstream linear-probe accuracy. Nothing in the definition of L_Reg is tied to the evaluation metric: the RBF bandwidth is set from batch statistics (sigma^2 = var(d_ij), following [25]), and the loss weight and layer choices are hyperparameters selected per dataset. The evaluation via frozen-encoder linear probing is external to the pretraining objective, so the reported accuracy is not a relabeling of the loss value. No load-bearing step reduces by construction to its inputs: the paper does not fit a parameter to a subset of data and then call the fitted value a prediction; lambda is swept in Table 4 and the chosen value (lambda=1) is then used for the reported runs. That selection is a legitimate concern about test-set tuning and reproducibility, as the ablation shows 69.0 vs 60.0 for lambda=1 vs 0.1, but it is a statistical/validation concern, not circularity. The layer-pair choice K={l,L} is motivated by experiments (Sec. 5.2) rather than derived, which is post-hoc selection rather than a definitional shortcut. The paper's explicit limitation that CNN gains are small does not undermine the independence of the method and its evaluation. There are no load-bearing self-citations and no imported uniqueness theorem. The result is therefore self-contained as an empirical contribution; circularity score 0.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The central method rests on a handful of domain assumptions about manifold geometry and layer similarity. The only true fitted knobs are the loss weight, warmup and duration epochs, and the layer pair selection. No new physical or structural entities are introduced.

free parameters (4)

lambda (regularization weight) = 1.0 for most MAE runs, tuned per dataset
Overall weight of LReg in the total loss. Table 4 shows strong sensitivity (69.0 at 1.0 vs 60.0 at 0.1). Tuned without a held-out validation set.
est (warmup epochs) = 10 for most datasets, 60 for CIFAR-100
Epoch at which the regularization is introduced. Tuned per dataset; the paper argues that a small est is best.
edur (duration epochs) = 100 for ImageNet-100 default; range 10-390 tested
Number of epochs the regularizer is active. The effect is modest (0.5% swings), but still a tuned parameter.
layer pair K (reference layer k and regularized layer l) = Penultimate layer as reference, last layer as regularized (e.g., {10, 11} for ViT-B/12)
Selected by experimentation as having 'maximum impact'. This is a fitted design choice that the paper does not derive.

assumptions (5)

domain assumption Manifold assumption: points close on the data manifold should have similar outputs.
Borrowed from Belkin et al. [4] and applied to intermediate latent spaces. The loss pulls together samples that are similar in the reference layer, assuming this indicates true similarity.
domain assumption Representation similarity in the reference layer k is a reliable proxy for semantic similarity.
Central to the method: if two images have close features in layer k, they should also be close in layer l. Stated in Section 3 but not proven or separately validated.
ad hoc to paper RBF kernel with sigma^2 = variance of pairwise distances is a stable and appropriate similarity measure.
Chosen following [25] for training stability. The choice affects the loss landscape; it is not derived from first principles.
standard math The normalized Laplacian formulation is algebraically equivalent to the original loss and improves stability.
The graph Laplacian identity is standard; used as a computational tool.
domain assumption ViT-based architectures lack the implicit regularization of CNNs, making them the right target for this loss.
Hypothesized in Section 4 to explain why gains are larger for MAE than for SimCLR/VICReg and why CNNs show smaller impact. Not mechanistically tested.

how reviews work

0 comments

Cite this review

Pith. "Pith review of MAGMA: Manifold Regularization for MAEs." pith.science (2026). https://pith.science/paper/VHYK4YM3

@misc{pith2026241202871,
  author       = {Pith},
  title        = {Pith review of: MAGMA: Manifold Regularization for MAEs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VHYK4YM3}},
  note         = {Machine review of arXiv:2412.02871}
}

read the original abstract

Masked Autoencoders (MAEs) are an important divide in self-supervised learning (SSL) due to their independence from augmentation techniques for generating positive (and/or negative) pairs as in contrastive frameworks. Their masking and reconstruction strategy also nicely aligns with SSL approaches in natural language processing. Most MAEs are built upon Transformer-based architectures where visual features are not regularized as opposed to their convolutional neural network (CNN) based counterparts, which can potentially hinder their performance. To address this, we introduce MAGMA, a novel batch-wide layer-wise regularization loss applied to representations of different Transformer layers. We demonstrate that by plugging in the proposed regularization loss, one can significantly improve the performance of MAE-based models. We further demonstrate the impact of the proposed loss on optimizing other generic SSL approaches (such as VICReg and SimCLR), broadening the impact of the proposed approach. Our code base can be found at https://github.com/adondera/magma.

Figures

Figures reproduced from arXiv: 2412.02871 by the authors.

**Figure 2.** (a) The regularization loss showed for MAE and M-MAE. For MAE we calculate the loss without backpropagating. For M-MAE, [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗

**Figure 3.** Effect of regularization. Implication: if the representa [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗

Figures from the paper (5 more)

**Figure 4.** Figure 4: PaCMAP plots for MAE-based methods. Applying [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]

**Figure 5.** Figure 5: Visualization of PCA’s leading component for features extracted from different layers of a ViT-B pretrained using MAE, M-MAE [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]

**Figure 6.** Figure 6: Attention maps from the 12 attention heads of the last layer of a ViT-B. The maps are extracted over the four MAE-based methods [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]

**Figure 1.** Figure 1: Additional visualization of PCA’s leading component for features extracted from different layers of a ViT-B pretrained using [PITH_FULL_IMAGE:figures/full_fig_p013_1.png]

**Figure 2.** Figure 2: Additional attention maps from the 12 attention heads of the last layer of a ViT-B. The attention maps come from three different [PITH_FULL_IMAGE:figures/full_fig_p014_2.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

39 extracted references · 18 canonical work pages

[1]

write newline

" write newline "" before.all 'output.state := FUNCTION fin.entry add.period write newline FUNCTION new.block output.state before.all = 'skip after.block 'output.state := if FUNCTION new.sentence output.state after.block = 'skip output.state before.all = 'skip after.sentence 'output.state := if if FUNCTION not #0 #1 if FUNCTION and 'skip pop #0 if FUNCTIO...
[2]

Deep vit features as dense visual descriptors

Shir Amir, Yossi Gandelsman, Shai Bagon, and Tali Dekel. Deep vit features as dense visual descriptors. arXiv preprint arXiv:2112.05814 , 2(3):4, 2021

arXiv 2021
[3]

Beit: Bert pre-training of image transformers

Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei. Beit: Bert pre-training of image transformers. arXiv preprint arXiv:2106.08254 , 2021

arXiv 2021
[4]

Vicreg: Variance-invariance-covariance regularization for self-supervised learning

Adrien Bardes, Jean Ponce, and Yann LeCun. Vicreg: Variance-invariance-covariance regularization for self-supervised learning. arXiv preprint arXiv:2105.04906 , 2021

arXiv 2021
[5]

Manifold regularization: A geometric framework for learning from labeled and unlabeled examples

Mikhail Belkin, Partha Niyogi, and Vikas Sindhwani. Manifold regularization: A geometric framework for learning from labeled and unlabeled examples. Journal of machine learning research , 7(11), 2006

work page 2006
[6]

Unsupervised learning of visual features by contrasting cluster assignments

Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. Advances in neural information processing systems , 33:9912--9924, 2020

work page 2020
[7]

Emerging properties in self-supervised vision transformers

Mathilde Caron, Hugo Touvron, Ishan Misra, Herv \'e J \'e gou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision , pages 9650--9660, 2021

2021
[8]

A simple framework for contrastive learning of visual representations

Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning , pages 1597--1607. PMLR, 2020

2020

Show all 39 references

[9]

Masked image modeling advances 3d medical image analysis

Zekai Chen, Devansh Agarwal, Kshitij Aggarwal, Wiem Safta, Mariann Micsinai Balan, and Kevin Brown. Masked image modeling advances 3d medical image analysis. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision , pages 1970--1980, 2023

1970
[10]

An analysis of single-layer networks in unsupervised feature learning

Adam Coates, Andrew Ng, and Honglak Lee. An analysis of single-layer networks in unsupervised feature learning. In Proceedings of the fourteenth international conference on artificial intelligence and statistics , pages 215--223. JMLR Workshop and Conference Proceedings, 2011

2011
[11]

A cluster separation measure

David L Davies and Donald W Bouldin. A cluster separation measure. IEEE transactions on pattern analysis and machine intelligence , (2):224--227, 1979

1979
[12]

Bert: Pre-training of deep bidirectional transformers for language understanding

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 , 2018

2018 arXiv
[13]

An image is worth 16x16 words: Transformers for image recognition at scale

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv...

2010 arXiv
[14]

Bootstrap your own latent-a new approach to self-supervised learning

Jean-Bastien Grill, Florian Strub, Florent Altch \'e , Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learning. Advances in neur...

2020
[15]

Masked autoencoders are scalable vision learners

Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Doll \'a r, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 16000--16009, 2022

2022
[16]

Robust object tracking using manifold regularized convolutional neural networks

Hongwei Hu, Bo Ma, Jianbing Shen, Hanqiu Sun, Ling Shao, and Fatih Porikli. Robust object tracking using manifold regularized convolutional neural networks. IEEE Transactions on Multimedia , 21(2):510--521, 2018

2018
[17]

Manifold regularized multitask feature learning for multimodality disease classification

Biao Jie, Daoqiang Zhang, Bo Cheng, Dinggang Shen, and Alzheimer's Disease Neuroimaging Initiative. Manifold regularized multitask feature learning for multimodality disease classification. Human brain mapping , 36(2):489--507, 2015

2015
[18]

Manifold regularization for locally stable deep neural networks

Charles Jin and Martin Rinard. Manifold regularization for locally stable deep neural networks. arXiv preprint arXiv:2003.04286 , 2020

2003 arXiv
[19]

A mutual information maximization perspective of language representation learning

Lingpeng Kong, Cyprien de Masson d'Autume, Wang Ling, Lei Yu, Zihang Dai, and Dani Yogatama. A mutual information maximization perspective of language representation learning. arXiv preprint arXiv:1910.08350 , 2019

1910 arXiv
[20]

Learning multiple layers of features from tiny images

Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009

2009
[21]

Tiny imagenet visual recognition challenge

Ya Le and Xuan Yang. Tiny imagenet visual recognition challenge. CS 231N , 7(7):3, 2015

2015
[22]

Contrastive tuning: A little help to make masked autoencoders forget

Johannes Lehner, Benedikt Alkin, Andreas F \"u rst, Elisabeth Rumetshofer, Lukas Miklautz, and Sepp Hochreiter. Contrastive tuning: A little help to make masked autoencoders forget. arXiv preprint arXiv:2304.10520 , 2023

2023 arXiv
[23]

Topic extraction from extremely short texts with variational manifold regularization

Ximing Li, Yang Wang, Jihong Ouyang, and Meng Wang. Topic extraction from extremely short texts with variational manifold regularization. Machine Learning , 110:1029--1066, 2021

2021
[24]

Umap: Uniform manifold approximation and projection for dimension reduction

Leland McInnes, John Healy, and James Melville. Umap: Uniform manifold approximation and projection for dimension reduction. arXiv preprint arXiv:1802.03426 , 2018

2018 arXiv
[25]

Masked autoencoders for point cloud self-supervised learning

Yatian Pang, Wenxiao Wang, Francis EH Tay, Wei Liu, Yonghong Tian, and Li Yuan. Masked autoencoders for point cloud self-supervised learning. In European conference on computer vision , pages 604--621. Springer, 2022

2022
[26]

Embedding propagation: Smoother manifold for few-shot classification

Pau Rodr \' guez, Issam Laradji, Alexandre Drouin, and Alexandre Lacoste. Embedding propagation: Smoother manifold for few-shot classification. In Computer Vision--ECCV 2020: 16th European Conference, Glasgow, UK, August 23--28, 2020, Proceedings, Part XXVI 16 , pages 121--138...

2020
[27]

Spectralnet: Spectral clustering using deep neural networks

Uri Shaham, Kelly Stanton, Henry Li, Boaz Nadler, Ronen Basri, and Yuval Kluger. Spectralnet: Spectral clustering using deep neural networks. arXiv preprint arXiv:1801.01587 , 2018

2018 arXiv
[28]

Contrastive multiview coding

Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding. In Computer Vision--ECCV 2020: 16th European Conference, Glasgow, UK, August 23--28, 2020, Proceedings, Part XI 16 , pages 776--794. Springer, 2020

2020
[29]

Manifold regularized deep neural networks

Vikrant Singh Tomar and Richard C Rose. Manifold regularized deep neural networks. In Fifteenth Annual Conference of the International Speech Communication Association , 2014

2014
[30]

Graph based manifold regularized deep neural networks for automatic speech recognition

Vikrant Singh Tomar and Richard C Rose. Graph based manifold regularized deep neural networks for automatic speech recognition. arXiv preprint arXiv:1606.05925 , 2016

2016 arXiv
[31]

Visualizing data using t-sne

Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research , 9(11), 2008

2008
[32]

Understanding how dimension reduction tools work: an empirical approach to deciphering t-sne, umap, trimap, and pacmap for data visualization

Yingfan Wang, Haiyang Huang, Cynthia Rudin, and Yaron Shaposhnik. Understanding how dimension reduction tools work: an empirical approach to deciphering t-sne, umap, trimap, and pacmap for data visualization. The Journal of Machine Learning Research , 22(1):9129--9201, 2021

2021
[33]

Simmim: A simple framework for masked image modeling

Zhenda Xie, Zheng Zhang, Yue Cao, Yutong Lin, Jianmin Bao, Zhuliang Yao, Qi Dai, and Han Hu. Simmim: A simple framework for masked image modeling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 9653--9663, 2022

2022
[34]

Refining word reesprentations by manifold learning

Chu Yonghe, Hongfei Lin, Liang Yang, Yufeng Diao, Shaowu Zhang, and Fan Xiaochao. Refining word reesprentations by manifold learning. In Proc. 28th Int. Joint Conf. Artif. Intell , pages 5394--5400, 2019

2019
[35]

Barlow twins: Self-supervised learning via redundancy reduction

Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and St \'e phane Deny. Barlow twins: Self-supervised learning via redundancy reduction. In International Conference on Machine Learning , pages 12310--12320. PMLR, 2021

2021
[36]

How mask matters: Towards theoretical understandings of masked autoencoders

Qi Zhang, Yifei Wang, and Yisen Wang. How mask matters: Towards theoretical understandings of masked autoencoders. Advances in Neural Information Processing Systems , 35:27127--27139, 2022

2022
[37]

ibot: Image bert pre-training with online tokenizer

Jinghao Zhou, Chen Wei, Huiyu Wang, Wei Shen, Cihang Xie, Alan Yuille, and Tao Kong. ibot: Image bert pre-training with online tokenizer. arXiv preprint arXiv:2111.07832 , 2021

2021 arXiv
[38]

Self pre-training with masked autoencoders for medical image analysis

Lei Zhou, Huidong Liu, Joseph Bae, Junjun He, Dimitris Samaras, and Prateek Prasanna. Self pre-training with masked autoencoders for medical image analysis. arXiv preprint arXiv:2203.05573 , 1(3), 2022

2022 arXiv
[39]

solo-learn: A library of self-supervised methods for visual representation learning

Victor Guilherme Turrisi da Costa, Enrico Fini, Moin Nabi, Nicu Sebe, and Elisa Ricci. solo-learn: A library of self-supervised methods for visual representation learning. Journal of Machine Learning Research , 23(56):1--6, 2022

2022

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.

[1] [1]

write newline

" write newline "" before.all 'output.state := FUNCTION fin.entry add.period write newline FUNCTION new.block output.state before.all = 'skip after.block 'output.state := if FUNCTION new.sentence output.state after.block = 'skip output.state before.all = 'skip after.sentence 'output.state := if if FUNCTION not #0 #1 if FUNCTION and 'skip pop #0 if FUNCTIO...

[2] [2]

Deep vit features as dense visual descriptors

Shir Amir, Yossi Gandelsman, Shai Bagon, and Tali Dekel. Deep vit features as dense visual descriptors. arXiv preprint arXiv:2112.05814 , 2(3):4, 2021

arXiv 2021

[3] [3]

Beit: Bert pre-training of image transformers

Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei. Beit: Bert pre-training of image transformers. arXiv preprint arXiv:2106.08254 , 2021

arXiv 2021

[4] [4]

Vicreg: Variance-invariance-covariance regularization for self-supervised learning

Adrien Bardes, Jean Ponce, and Yann LeCun. Vicreg: Variance-invariance-covariance regularization for self-supervised learning. arXiv preprint arXiv:2105.04906 , 2021

arXiv 2021

[5] [5]

Manifold regularization: A geometric framework for learning from labeled and unlabeled examples

Mikhail Belkin, Partha Niyogi, and Vikas Sindhwani. Manifold regularization: A geometric framework for learning from labeled and unlabeled examples. Journal of machine learning research , 7(11), 2006

work page 2006

[6] [6]

Unsupervised learning of visual features by contrasting cluster assignments

Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. Advances in neural information processing systems , 33:9912--9924, 2020

work page 2020

[7] [7]

Emerging properties in self-supervised vision transformers

Mathilde Caron, Hugo Touvron, Ishan Misra, Herv \'e J \'e gou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision , pages 9650--9660, 2021

2021

[8] [8]

A simple framework for contrastive learning of visual representations

Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning , pages 1597--1607. PMLR, 2020

2020