Pith. sign in

REVIEW 4 major objections 5 minor 54 references

GViT: Representing Images as Gaussians for Visual Recognition

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read GViT shows a ViT can classify from a few hundred 2D Gaussians, reaching 76.9% top-1 on ImageNet-1k, within 1.8 points of a patch-based ViT-B/16.

desk verdict A novel Gaussian-token representation with a real ImageNet result, but the guidance ablation is confounded and the theory section needs a rewrite. read the letter →

arxiv 2506.23532 v1 pith:6SYGPAZ3 submitted 2025-06-30 cs.CV cs.LG

Jefferson Hernandez , Ruozhen He , Guha Balakrishnan , Alexander C. Berg , Vicente Ordonez This is my paper · ORCID
classification cs.CVcs.LG
keywords 2DGaussianrepresentationvisiontransformerimageclassificationdifferentiablerenderinggradientguidanceinterpretabilitynon-pixelinputsNet-1k
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that image classification does not need pixels or a patch grid: a standard Vision Transformer can instead take as input a few hundred learnable 2D Gaussians, each defined by center, scale, orientation, color, and opacity, and still perform almost as well as a conventional patch-based ViT. The proposed GViT pipeline learns the Gaussian placement jointly with the classifier, using the classifier's own gradients to steer Gaussians toward class-salient regions while a differentiable renderer enforces reconstruction fidelity. The headline result is 76.9% top-1 accuracy on ImageNet-1k with a ViT-B backbone, only 1.8 points below a regular ViT-B/16 (78.7%) and above other non-pixel representations such as JPEG-S and Perceiver-IO. If the finding holds, it suggests that explicit mid-level primitives are a viable, interpretable middle ground between raw pixels and learned patches.

What carries the argument

The load-bearing object is a set of $k$ 2D Gaussian primitives, each a 9-dimensional vector $g = (p, s, \phi, r, o)$ holding center, scale, 2D rotation angle, color, and opacity, rendered into an image by a differentiable Gaussian splatter. A denoising encoder predicts residuals on randomly initialized Gaussians, and a ViT classifier consumes the Gaussian tokens directly. The mechanism that makes the representation task-adapted is 'constructive FGSM' guidance: during joint training, the update to the Gaussian parameters is $\nabla_\theta (L_{\mathrm{pix}} + \lambda L_{\mathrm{perc}}) - \gamma \nabla_\theta L_{\mathrm{cls}}$, so the classifier's cross-entropy gradient (normally used to attack) is applied with the opposite sign to pull primitives toward class evidence while reconstruction losses preserve image fidelity. The training is organized in a three-phase schedule—reconstruction warm-up, classifier pre-training with frozen Gaussians, then joint optimization with guidance—and the paper reports that this guidance is what relocates Gaussians from a quasi-grid into class-salient clusters.

What would settle it

Train GViT-B on ImageNet-1k with the same schedule but flip the sign of the guidance term (add $+\gamma \nabla_\theta L_{\mathrm{cls}}$ instead of subtracting it); if top-1 accuracy does not fall by roughly 3 points relative to the guided model, the constructive-guidance explanation for the gain is wrong.

Watch

Extended reading notes

Core claim

The central claim is that, with a compact set of 2D Gaussians as input and a classifier-gradient guidance term added to the reconstruction update, a relatively standard ViT matches patch-based performance closely: GViT-B reaches 76.9% top-1 on ImageNet-1k versus 78.7% for ViT-B/16. Ablations attribute 3.3 points of that score to guidance: without the gradient term the same model drops to 73.6%. The paper further shows the guided representation outperforms other non-pixel inputs (JPEG-S 76.5%, Perceiver-IO 72.7%), transfers to fine-grained benchmarks by fine-tuning only the classifier head, and that the determinant of each Gaussian's covariance clusters in class-discriminative regions, offering a natural interpretability signal. The authors frame the contribution as an empirical existence proof that explicit mid-level Gaussian representations are viable, interpretable, and surprisingly strong, and state explicitly that patch-based ViTs remain the pragmatic choice for large-scale deployment.

Load-bearing premise

The headline result rests on the assumption that the classifier-gradient guidance term is a reliable, transferable signal that genuinely relocates Gaussians to class-salient regions and is the cause of the measured 3.3-point accuracy gain, rather than a side effect such as extra noise or an implicit regularizer.

Editorial extensions

If this is right

  • GViT-B reaches 76.9% top-1 on ImageNet-1k, within 1.8 points of a patch-based ViT-B/16, using only hundreds of Gaussians per image as input.
  • Removing the guidance term costs 3.3 points (76.9% to 73.6%), so the classifier-gradient steering, not just the Gaussian representation, is doing much of the work.
  • On transfer benchmarks, freezing the ImageNet-trained Gaussian encoder and fine-tuning only the classifier head improves average accuracy from 81.0% to 83.6% when guidance is used.
  • GViT-B's 76.9% exceeds other non-pixel-input approaches, including JPEG-S (76.5%) and Perceiver-IO (72.7%), while remaining under the patch-based ViT-B/16 (78.7%).
  • Gaussians trained with guidance concentrate in class-salient regions, and the determinant of their covariance serves as a gradient-free interpretability map that aligns with class-discriminative attention.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural extension is to apply the same 'negative classifier gradient' update to other differentiable structured representations, such as superpixels, slot decompositions, or 3D Gaussian scenes, turning adversarial gradients into a general representation-shaping tool.
  • The 2–3 point gap between classifiers trained on parametric Gaussians versus rasterized renderings suggests the parameters themselves carry usable information beyond the pixels they produce; a next test is whether this is a geometric inductive bias or an artifact that disappears under stronger regularization.
  • Because accuracy improves monotonically with Gaussian count up to roughly 768 and then saturates, an adaptive representation that dynamically splits or merges Gaussians during training could plausibly push the Pareto frontier further than a fixed budget.
  • A testable ablation is to compare the proposed guidance gradient against a random or fixed direction of the same magnitude, which would isolate whether the class-gradient direction itself helps or merely adds beneficial stochasticity.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces GViT, a classification framework that replaces pixel or patch inputs with a compact set of learnable 2D Gaussians. A denoising Gaussian encoder predicts Gaussian parameters (position, scale, orientation, color, opacity) from image patches; these are rendered with a differentiable Gaussian splatter, and a ViT classifier consumes the Gaussian tokens directly. The training procedure includes a 'constructive guidance' step in which classifier gradients are added to the reconstruction update to steer Gaussians toward class-salient regions. On ImageNet-1k, GViT-B with guidance reaches 76.9% top-1 accuracy, 1.8 points below a patch-based ViT-B/16 (78.7%), and outperforms several non-pixel representations such as JPEG-S (76.5%) and Perceiver-IO (72.7%). The paper claims that removing guidance drops GViT-B to 73.6%, attributing 3.3 points to the guidance mechanism. Additional experiments on transfer learning and interpretability (covariance-based saliency and CDAM) are reported.

Significance. If the empirical claim holds, the paper demonstrates that an explicit mid-level representation based on 2D Gaussians can support competitive image classification, which would be a notable result in the broader effort to move beyond pixel-grid inputs. The comparison against external baselines (JPEG-S, Perceiver-IO, Pixel-ViT) is useful, and the leakage-avoidance experiments are a reasonable attempt to rule out hidden cues. The paper also provides ablations on the number of Gaussians, scale cap, guidance strength, and loss function. However, the central guidance mechanism is not cleanly isolated: the 'no-guidance' condition in the ablation still provides full classifier gradients to the encoder, and the theoretical derivation in Section 4 contains algebraic errors. The significance of the 3.3-point guidance gain, which the paper emphasizes as the key contribution, is therefore not established. The underlying representation result is plausible, but the paper requires substantial correction before its central contribution can be accepted.

major comments (4)
  1. [Section 4, 'Constructive FGSM in parameter space'] The derivation of the L∞-constrained update is incorrect. The problem is to maximize ∇θH(θ;w)ᵀΔθ subject to ||Δθ||∞ ≤ ε, and the maximizer is ε·sign(∇θH), not ε·sign(w). The paper states Δθ* = ε·sign(w) and then concludes the logit increase is ε||w||₁. This is algebraically wrong: even if sign(w) were the correct direction, the logit increase from Δθ = ε·sign(w) is wᵀΔθ = ε||w||₁, but the maximizer of the linearized loss is sign(∇θH), not sign(w). Moreover, the claim that ||w||₁ grows linearly with d = 9k is not generally valid because the classifier weights are learned and may be sparse or norm-bounded. Consequently, the theoretical justification for why guidance helps, the 'dimensionality argument', is not supported.
  2. [Section 2.2, Eq. (1) and Appendix A, Algorithm 1] The equations and pseudocode are mutually inconsistent. Equation (1) writes the composite update as ∇θ(Lpix + λpercLperc) − γ∇θLcls. Under a standard descent step θ ← θ − η·tilde_grad, this would move θ in the direction +γ∇θLcls, which increases the cross-entropy loss rather than decreasing it. The pseudocode instead sets gauss_encoder.grad = rec_grads + γ * cls_grads and then calls opt_enc.step(), which implements θ ← θ − η(rec_grads + γ cls_grads), the opposite sign from Eq. (1). In addition, the pseudocode reads gauss_cls.grad after executing gauss_cls.requires_grad_(False); at that point the classifier's parameters do not require gradients, so gauss_cls.grad is not populated (the gradient of loss_cls with respect to the encoder parameters would be in gauss_encoder's parameter grads). The method as specified is therefore ambiguous and not directly reproducible.
  3. [Section 3.1 and Table 2, 'Guidance vs. no guidance' ablation] The ablation that attributes 3.3 points to guidance is confounded. Algorithm 1 computes loss = loss_pixel + λperc·loss_perc + λcls·loss_cls on every normal training step and calls loss.backward() followed by opt_enc.step(), so the encoder receives the classifier gradient at full strength (λcls = 1) during the joint training phase. The no-guidance schedule is described in the appendix as 'the same as before but with no guidance added for the last 50 epochs', meaning the only difference is that the guided condition replaces some normal steps with a composite step whose classifier-gradient coefficient is γ = 0.1 instead of 1.0. Thus the no-guidance condition still supplies the encoder with substantial classification-gradient signal, and the observed 3.3-point gap may be caused by the different update structure (e.g., the freezing of the classifier during the composite step) rather than by the proposed 'constructive' direction. To support the paper's central claim, the authors must provide a control in which the encoder is detached from Lcls (or λcls is set to zero for the encoder) throughout, or otherwise show that the classifier gradient is truly absent from the no-guidance condition.
  4. [Section 4, Eq. (2) and the linearized analysis] Equation (2) replaces ∇θLcls with the classifier weight vector w, but the gradient of the cross-entropy loss with respect to the Gaussian parameters is not simply w; it is (σ(s(θ)) − y)·w in the linearized model s(θ) = wᵀθ. The paper does not justify this substitution, and the subsequent claim that 'setting γ > 0 ensures a descent step for cross-entropy' is not proven: the change in logit after the update θ − η(∇Lrec − γw) depends on the inner product between w and ∇Lrec, which can have either sign, so the inequality Lcls(θ − ηtilde_grad) < Lcls(θ) is not guaranteed without additional assumptions. This weakens the theoretical grounding of the guidance mechanism beyond the algebraic error in the FGSM derivation.
minor comments (5)
  1. [Section 3.1, 'Guidance coefficient' vs. Table 1b] The numbers in the text do not match Table 1b. The text reports 71.2%, 72.6%, 73.3%, and 74.8% for γ = 0.025, 0.05, 0.075, and 0.10, whereas Table 1b lists 72.3, 73.6, 74.2, and 75.2. Please correct the inconsistency.
  2. [Section 3.1, 'Gaussian Fitting' vs. 'Guidance coefficient'] The text in the 'Gaussian Fitting' paragraph states that the full gradient guidance delivers 75.2% top-1, while the 'Guidance coefficient' paragraph says the best result is 74.8% at γ = 0.10. These should be reconciled.
  3. [Section 3.2, cross-training numbers vs. Table 4] The text reports 67.4% when a classifier trained on SGD Gaussians is evaluated on learned Gaussians, and 68.5% when a classifier trained on learned Gaussians is evaluated on SGD Gaussians. Table 4 lists different numbers (e.g., 'Trained on Learned Gaussians' is 67.6 GS and 65.0 IMG). Please align the text with the table.
  4. [Section 3, architecture and experiments] The number of Gaussians used for the ImageNet-1k results in Table 2 is not stated. The paper mentions an inability to scale to more than 512 Gaussians on ImageNet, but Figure 3 is on Mini-ImageNet and uses up to 1024 Gaussians. Please specify the Gaussian count for all reported ImageNet results.
  5. [Appendix A, Algorithm 1] There are small naming inconsistencies in the pseudocode: 'squats' appears to be a typo for 'quats', 'loss_pix' and 'loss_pixel' are used interchangeably, and the variables plambda and clambda are defined but clambda is not used in the composite step. These should be cleaned up for reproducibility.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the headline accuracy is an externally benchmarked empirical result, and the guidance contribution is ablation-measured rather than defined into existence.

full rationale

GViT's central claim—that a compact set of 2D Gaussians, steered by classifier gradients, reaches 76.9% top-1 on ImageNet-1k—is an experimental result benchmarked against independently reported baselines (ViT-B/16, JPEG-S, Perceiver-IO, Pixel-ViT-B/1). The guidance contribution is isolated by explicit guided/no-guidance rows in Table 2 and by the Mini-IN-100 ablations in Table 1d; the guidance coefficient gamma is swept in Table 1b and then transferred to ImageNet, which is ordinary hyperparameter selection, not a fitted input renamed as a prediction. Eq. (1) and the three-phase schedule define the training objective rather than assuming the outcome, and the paper's own limitations section acknowledges sensitivity to the guidance coefficient and schedule. The suspicious parts of the manuscript—the informal linearized argument in Section 4 and the discrepancies between Eq. (1), Eq. (2), and Algorithm 1's pseudocode—concern mathematical rigor and experimental control, not circularity: I cannot exhibit any equation or citation through which the reported accuracies reduce by construction to the loss definitions, the hyperparameters, or the authors' own prior work. There are no load-bearing self-citations, so the circularity score is 0.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central result rests on a handful of hand-set constants (Gaussian count, scale cap, guidance weight, perceptual loss weight) and on four unverified modeling premises: the renderer's fidelity, the guidance signal's effectiveness, the ViT's compatibility with tokenized Gaussians, and the interpretation of the leakage experiments. None of these is derived or externally benchmarked beyond the paper's own ablations.

free parameters (4)
  • Number of Gaussians k = 512 for ImageNet-1k; 64 to 1024 in ablations
    Fixed budget per image, selected from Fig. 3 ablation; ImageNet limited to 512 due to rendering memory.
  • Scale cap c = 1.0
    Bound on Gaussian size via c * sigmoid(s); Table 1a sweep on Mini-ImageNet selects c=1.0.
  • Guidance coefficient gamma = 0.1 (ViT-S), 0.05 (ViT-B transfer)
    Weight on the opposite-sign classification gradient in Eq. (1); Table 1b sweep selects 0.1, lowered to 0.05 for ViT-B stability.
  • Perceptual loss weight lambda_perc = 0.1
    Weight on DSSIM in the reconstruction loss, introduced at epoch 150; from Appendix Table A.1.
assumptions (4)
  • domain assumption Orthographic splatting of a fixed set of 2D Gaussians is a faithful, differentiable image model for classification.
    Section 2.1 defines the renderer and uses its gradients for both losses; no validation of the fidelity or sufficiency of this approximation is given.
  • domain assumption Negated classifier gradients added to the encoder update steer Gaussians toward class-salient regions without harming reconstruction.
    Section 2.2, Eq. (1), and the three-phase schedule; the paper's theoretical justification (Section 4) is algebraically flawed, so this remains an assumption.
  • domain assumption A standard ViT can consume a set of 9-dimensional Gaussian parameter vectors as input tokens.
    Section 2.2 feeds the k Gaussian latents to an MLP and then to a ViT; no positional structure beyond the tokens is provided, and this is not explicitly validated.
  • ad hoc to paper The leakage tests' interpretation: a 2-3 point drop when rendering to images is evidence against hidden cues.
    Section 3.2 interprets small accuracy drops as ruling out 'cheat codes'; without a baseline for expected re-encoding loss, this conclusion is not forced.

how reviews work

0 comments
Cite this review

Pith. "Pith review of GViT: Representing Images as Gaussians for Visual Recognition." pith.science (2026). https://pith.science/paper/6SYGPAZ3

@misc{pith2026250623532,
  author       = {Pith},
  title        = {Pith review of: GViT: Representing Images as Gaussians for Visual Recognition},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6SYGPAZ3}},
  note         = {Machine review of arXiv:2506.23532}
}
read the original abstract

We introduce GVIT, a classification framework that abandons conventional pixel or patch grid input representations in favor of a compact set of learnable 2D Gaussians. Each image is encoded as a few hundred Gaussians whose positions, scales, orientations, colors, and opacities are optimized jointly with a ViT classifier trained on top of these representations. We reuse the classifier gradients as constructive guidance, steering the Gaussians toward class-salient regions while a differentiable renderer optimizes an image reconstruction loss. We demonstrate that by 2D Gaussian input representations coupled with our GVIT guidance, using a relatively standard ViT architecture, closely matches the performance of a traditional patch-based ViT, reaching a 76.9% top-1 accuracy on Imagenet-1k using a ViT-B architecture.

Figures

Figures reproduced from arXiv: 2506.23532 by the authors.

Figure 1
Figure 1. The original Vision Transformer (ViT) [9] model uses a grid of contiguous patch inputs as a representation. Alternative models have explored pixel-level representations [23, 21, 36]. Our work explores the use of Gaussians as our basic primitive. To make training practical in Gaussian space, we train a model that performs Gaussian fitting in a single forward pass as opposed to the iterative optimization method more c… view at source ↗
Figure 2
Figure 2. Overview of the training pipeline for GViT. Our model is iteratively trained to recognize [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Number of Gaussians: Mini-Imagenet classification performance with 64, 128, 256, 512, 768, and 1024 Gaussians per image. As we increase the number of Gaussians, the performance increases monotonically [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Visualization of Gaussians for a sample image under three training regimes: (a) The [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: Gradient-guided Gaussians exhibit a natural interpretability component. (a) Original [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

54 extracted references · 34 canonical work pages

  1. [1]

    Slic superpixels compared to state-of-the-art superpixel methods

    Radhakrishna Achanta, Appu Shaji, Kevin Smith, Aurélien Lucchi, Pascal Fua, and Sabine Süsstrunk. Slic superpixels compared to state-of-the-art superpixel methods. IEEE Transactions on Pattern Analysis and Machine Intelligence, 34(11):2274–2282, 2012

    work page 2012

  2. [2]

    Beit: Bert pre-training of image trans- formers

    Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei. Beit: Bert pre-training of image trans- formers. arXiv preprint arXiv:2106.08254, 2021

    arXiv 2021

  3. [3]

    Ondrej Biza, Sjoerd Van Steenkiste, Mehdi S. M. Sajjadi, Gamaleldin Fathy Elsayed, Aravindh Mahendran, and Thomas Kipf. Invariant slot attention: Object discovery with slot-centric reference frames. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, Proceedings of the 40th International Conf...

    work page 2023

  4. [4]

    Class-discriminative attention maps for vision transformers

    Lennart Brocki, Jakub Binda, and Neo Christopher Chung. Class-discriminative attention maps for vision transformers. arXiv preprint arXiv:2312.02364, 2023

    arXiv 2023

  5. [5]

    Generative pretraining from pixels

    Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, and Ilya Sutskever. Generative pretraining from pixels. In International conference on machine learning, pages 1691–1703. PMLR, 2020

    2020

  6. [6]

    Electra: Pre-training text encoders as discriminators rather than generators

    Kevin Clark, Minh-Thang Luong, Quoc V Le, and Christopher D Manning. Electra: Pre-training text encoders as discriminators rather than generators. arXiv preprint arXiv:2003.10555, 2020

    arXiv 2003

  7. [7]

    Vision transformers need registers

    Timothée Darcet, Maxime Oquab, Julien Mairal, and Piotr Bojanowski. Vision transformers need registers. In The Twelfth International Conference on Learning Representations, 2024

    work page 2024

  8. [8]

    Scaling vision transformers to 22 billion parameters

    Mostafa Dehghani, Josip Djolonga, Basil Mustafa, Piotr Padlewski, Jonathan Heek, Justin Gilmer, Andreas Peter Steiner, Mathilde Caron, Robert Geirhos, Ibrahim Alabdulmohsin, et al. Scaling vision transformers to 22 billion parameters. In International Conference on Machine Learning, pages 7480–7512. PMLR, 2023

    2023

Show all 54 references
  1. [9]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, G Heigold, S Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on...

    2020

  2. [10]

    Adaptive slot attention: Object discovery with dynamic slot number

    Ke Fan, Zechen Bai, Tianjun Xiao, Tong He, Max Horn, Yanwei Fu, Francesco Locatello, and Zheng Zhang. Adaptive slot attention: Object discovery with dynamic slot number. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 23062–23071, 2024

    2024

  3. [11]

    3d gaussian splatting as new era: A survey

    Ben Fei, Jingyi Xu, Rui Zhang, Qingyuan Zhou, Weidong Yang, and Ying He. 3d gaussian splatting as new era: A survey. IEEE Transactions on Visualization and Computer Graphics, 2024

    2024

  4. [12]

    Efficient graph-based image segmentation

    Pedro F Felzenszwalb and Daniel P Huttenlocher. Efficient graph-based image segmentation. International journal of computer vision, 59:167–181, 2004

    2004

  5. [13]

    Understanding the difficulty of training deep feedfor- ward neural networks

    Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedfor- ward neural networks. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pages 249–256. JMLR Workshop and Conference Proceedings, 2010

    2010

  6. [14]

    Explaining and harnessing adversar- ial examples

    Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversar- ial examples. arXiv preprint arXiv:1412.6572, 2014

    2014 arXiv

  7. [15]

    Accurate, large minibatch sgd: Training imagenet in 1 hour

    Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large minibatch sgd: Training imagenet in 1 hour. arXiv preprint arXiv:1706.02677, 2017

    2017 arXiv

  8. [16]

    Faster neural networks straight from jpeg

    Lionel Gueguen, Alex Sergeev, Ben Kadlec, Rosanne Liu, and Jason Yosinski. Faster neural networks straight from jpeg. Advances in Neural Information Processing Systems, 31, 2018. 10

    2018

  9. [17]

    Masked autoencoders are scalable vision learners

    Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 16000–16009, 2022

    2022

  10. [18]

    Deep residual learning for im- age recognition

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for im- age recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), CVPR ’16, pages 770–778, June 2016

    2016

  11. [19]

    Bytes are all you need: Transformers operating directly on file bytes

    Maxwell Horton, Sachin Mehta, Ali Farhadi, and Mohammad Rastegari. Bytes are all you need: Transformers operating directly on file bytes. Transactions of Machine Learning Research (TMLR), 2024

    2024

  12. [20]

    2d gaussian splatting for geometrically accurate radiance fields

    Binbin Huang, Zehao Yu, Anpei Chen, Andreas Geiger, and Shenghua Gao. 2d gaussian splatting for geometrically accurate radiance fields. In ACM SIGGRAPH 2024 conference papers, pages 1–11, 2024

    2024

  13. [21]

    Perceiver IO: A General Architecture for Structured Inputs & Outputs

    Andrew Jaegle, Sebastian Borgeaud, Jean-Baptiste Alayrac, Carl Doersch, Catalin Ionescu, David Ding, Skanda Koppula, Daniel Zoran, Andrew Brock, Evan Shelhamer, et al. Perceiver IO: A General Architecture for Structured Inputs & Outputs. In International Conference on Learning...

    2022

  14. [22]

    Perceiver IO: A general architecture for structured inputs & outputs

    Andrew Jaegle, Sebastian Borgeaud, Jean-Baptiste Alayrac, Carl Doersch, Catalin Ionescu, David Ding, Skanda Koppula, Daniel Zoran, Andrew Brock, Evan Shelhamer, Olivier J Henaff, Matthew Botvinick, Andrew Zisserman, Oriol Vinyals, and Joao Carreira. Perceiver IO: A general arc...

    2022

  15. [23]

    Perceiver: General Perception with Iterative Attention

    Andrew Jaegle, Felix Gimeno, Andy Brock, Oriol Vinyals, Andrew Zisserman, and Joao Carreira. Perceiver: General Perception with Iterative Attention. In International conference on machine learning, pages 4651–4664. PMLR, 2021

    2021

  16. [24]

    Superpixel sampling networks

    Varun Jampani, Deqing Sun, Ming-Yu Liu, Ming-Hsuan Yang, and Jan Kautz. Superpixel sampling networks. In Proceedings of the European Conference on Computer Vision (ECCV), pages 352–368, 2018

    2018

  17. [25]

    Unsupervised image segmentation by backpropagation

    Asako Kanezaki. Unsupervised image segmentation by backpropagation. In 2018 IEEE international conference on acoustics, speech and signal processing (ICASSP), pages 1543–

    2018

  18. [26]

    3d gaussian splatting for real-time radiance field rendering

    Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 3d gaussian splatting for real-time radiance field rendering. ACM Transactions on Graphics, 42(4):1–14, 2023

    2023

  19. [27]

    Seac and the start of image processing at the national bureau of standards

    Russell A Kirsch. Seac and the start of image processing at the national bureau of standards. IEEE Annals of the History of Computing, 20(2):7–13, 1998

    1998

  20. [28]

    Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. Imagenet classification with deep convolutional neural networks. In F. Pereira, C. J. C. Burges, L. Bottou, and K. Q. Weinberger, editors, Advances in Neural Information Processing Systems 25 , pages 1097–1105. Curran As...

    2012

  21. [29]

    Kutulakos, David J

    Alex Levinshtein, Adrian Stere, Kiriakos N. Kutulakos, David J. Fleet, Sven J. Dickinson, and Kaleem Siddiqi. Turbopixels: Fast superpixels using geometric flows. IEEE Transactions on Pattern Analysis and Machine Intelligence, 31(12):2290–2302, 2009

    2009

  22. [30]

    Pytorch distributed: Experiences on accelerating data parallel training

    Shen Li, Yanli Zhao, Rohan Varma, Omkar Salpekar, Pieter Noordhuis, Teng Li, Adam Paszke, Jeff Smith, Brian Vaughan, Pritam Damania, et al. Pytorch distributed: Experiences on accelerating data parallel training. arXiv preprint arXiv:2006.15704, 2020

    2006 arXiv

  23. [31]

    A convnet for the 2020s

    Zhuang Liu, Hanzi Mao, Chao-Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 11976–11986, June 2022. 11

    2022

  24. [32]

    Object-centric learning with slot attention

    Francesco Locatello, Dirk Weissenborn, Thomas Unterthiner, Aravindh Mahendran, Georg Heigold, Jakob Uszkoreit, Alexey Dosovitskiy, and Thomas Kipf. Object-centric learning with slot attention. Advances in neural information processing systems, 33:11525–11538, 2020

    2020

  25. [33]

    Sgdr: Stochastic gradient descent with warm restarts

    Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. In International Conference on Learning Representations, 2016

    2016

  26. [34]

    Decoupled weight decay regularization

    Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017

    2017 arXiv

  27. [35]

    Enhance the visual representation via discrete adversarial training

    Xiaofeng Mao, Yuefeng Chen, Ranjie Duan, Yao Zhu, Gege Qi, Shaokai Ye, Xiaodan Li, Rong Zhang, and Hui Xue. Enhance the visual representation via discrete adversarial training. arXiv preprint arXiv:2209.07735, 2022

    2022 arXiv

  28. [36]

    An image is worth more than 16x16 patches: Exploring transformers on individual pixels

    Duy-Kien Nguyen, Mahmoud Assran, Unnat Jain, Martin R Oswald, Cees GM Snoek, and Xinlei Chen. An image is worth more than 16x16 patches: Exploring transformers on individual pixels. International Conference on Learning Representations (ICLR), 2025

    2025

  29. [37]

    Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy V . V o, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel HAZIZA, Francisco Massa, Alaaeldin El-Nouby, Mido Assran, Nicolas Ballas, Wojciech Galuba, Russell Howes, Po-Yao Huang, Shang-Wen Li, Ishan Misra, Michael Rab...

    2024

  30. [38]

    Rgb no more: Minimally-decoded jpeg vision transformers

    Jeongsoo Park and Justin Johnson. Rgb no more: Minimally-decoded jpeg vision transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 22334–22346, 2023

    2023

  31. [39]

    Gaussian masked autoencoders

    Jathushan Rajasegaran, Xinlei Chen, Rulilong Li, Christoph Feichtenhofer, Jitendra Malik, and Shiry Ginosar. Gaussian masked autoencoders. arXiv preprint arXiv:2501.03229, 2025

    2025 arXiv

  32. [40]

    Learning a classification model for segmentation

    Xiaofeng Ren and Jitendra Malik. Learning a classification model for segmentation. In Proceedings Ninth IEEE International Conference on Computer Vision (ICCV), pages 10–17. IEEE, 2003

    2003

  33. [41]

    Very deep convolutional networks for large-scale image recognition

    Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. CoRR, abs/1409.1556, 2014

    2014 arXiv

  34. [42]

    How to train your vit? data, augmentation, and regularization in vision transformers

    Andreas Peter Steiner, Alexander Kolesnikov, Xiaohua Zhai, Ross Wightman, Jakob Uszkoreit, and Lucas Beyer. How to train your vit? data, augmentation, and regularization in vision transformers. Transactions on Machine Learning Research, 2022

    2022

  35. [43]

    Superpixels: An evaluation of the state-of-the-art

    David Stutz, Alexander Hermans, and Bastian Leibe. Superpixels: An evaluation of the state-of-the-art. Computer Vision and Image Understanding, 166:1–27, 2018

    2018

  36. [44]

    Single-view view synthesis with multiplane images

    Richard Tucker and Noah Snavely. Single-view view synthesis with multiplane images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 551–560, 2020

    2020

  37. [45]

    Stacked denoising autoencoders: Learning useful representations in a deep network with a local denoising criterion

    Pascal Vincent, Hugo Larochelle, Isabelle Lajoie, Yoshua Bengio, Pierre-Antoine Manzagol, and Léon Bottou. Stacked denoising autoencoders: Learning useful representations in a deep network with a local denoising criterion. Journal of machine learning research, 11(12), 2010

    2010

  38. [46]

    Matching networks for one shot learning

    Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, et al. Matching networks for one shot learning. Advances in neural information processing systems, 29, 2016

    2016

  39. [47]

    Image quality assessment: from error visibility to structural similarity

    Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing, 13(4):600– 612, 2004

    2004

  40. [48]

    Beyond language models: Byte models are digital world simulators

    Shangda Wu, Xu Tan, Zili Wang, Rui Wang, Xiaobing Li, and Maosong Sun. Beyond language models: Byte models are digital world simulators. arXiv preprint arXiv:2402.19155, 2024. 12

    2024 arXiv

  41. [49]

    Learning in the frequency domain

    Kai Xu, Minghai Qin, Fei Sun, Yuhao Wang, Yen-Kuang Chen, and Fengbo Ren. Learning in the frequency domain. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1740–1749, 2020

    2020

  42. [50]

    gsplat: An open-source library for gaussian splatting

    Vickie Ye, Ruilong Li, Justin Kerr, Matias Turkulainen, Brent Yi, Zhuoyang Pan, Otto Seiskari, Jianbo Ye, Jeffrey Hu, Matthew Tancik, and Angjoo Kanazawa. gsplat: An open-source library for gaussian splatting. Journal of Machine Learning Research, 26(34):1–17, 2025

    2025

  43. [51]

    Vector-quantized image modeling with improved vqgan

    Jiahui Yu, Xin Li, Jing Yu Koh, Han Zhang, Ruoming Pang, James Qin, Alexander Ku, Yuanzhong Xu, Jason Baldridge, and Yonghui Wu. Vector-quantized image modeling with improved vqgan. arXiv preprint arXiv:2110.04627, 2021

    2021 arXiv

  44. [52]

    A survey on masked autoencoder for self-supervised learning in vision and beyond

    Chaoning Zhang, Chenshuang Zhang, Junha Song, John Seon Keun Yi, Kang Zhang, and In So Kweon. A survey on masked autoencoder for self-supervised learning in vision and beyond. arXiv preprint arXiv:2208.00173, 2022

    2022 arXiv

  45. [53]

    Gaussianimage: 1000 fps image representation and compression by 2d gaussian splatting

    Xinjie Zhang, Xingtong Ge, Tongda Xu, Dailan He, Yan Wang, Hongwei Qin, Guo Lu, Jing Geng, and Jun Zhang. Gaussianimage: 1000 fps image representation and compression by 2d gaussian splatting. In European Conference on Computer Vision, pages 327–345. Springer, 2024

    2024

  46. [54]

    Self-supervised learning of object parts for semantic segmentation

    Adrian Ziegler and Yuki M Asano. Self-supervised learning of object parts for semantic segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 14502–14511, 2022. 13 A Implementation Details Algorithm 1: GViT PyTorch pseudocode....

    2022

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.