Pith. sign in

REVIEW 3 major objections 5 minor 1 cited by

Quick ViTs: Speeding up Vision Transformers through Equivariance

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that enforcing octic (D8) equivariance in Vision Transformer linear layers cuts FLOPs by 5.33x per layer and about 40% for whole ViT-H models while matching ImageNet-1K accuracy.

desk verdict Genuine D8 extension with correct FLOP math, but the 'match baseline' claim needs retrained baselines and seeds. read the letter →

arxiv 2505.15441 v5 pith:IJ4D7AKK submitted 2025-05-21 cs.CV cs.AIcs.LG

David Nordström , Johan Edstedt , Fredrik Kahl , Georg Bökman This is my paper
classification cs.CVcs.AIcs.LG
keywords visiontransformerequivariancedihedralgroupD8FourierdomainefficientdeeplearningImageNetself-supervised
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to show that the geometric symmetries of natural images—90-degree rotations and reflections—can be built into Vision Transformers as exact octic ($D_8$) equivariance, and that doing so cuts computation without sacrificing accuracy. Its central claim is that an octic-equivariant linear layer uses 5.33x fewer FLOPs and up to 8x less memory per feature dimension than an ordinary linear layer, which translates to roughly 40% fewer FLOPs for a ViT-H model at matching ImageNet top-1 accuracy. The authors train supervised (DeiT III) and self-supervised (DINOv2) octic ViTs and report that the hybrid models H8 and I8 match or exceed the baseline on ImageNet-1K and several transfer tasks, while the fully equivariant $D_8$ model loses accuracy. If the claim is right, $D_8$ equivariance becomes a practical efficiency tool for large ViTs rather than a theoretical nicety.

What carries the argument

The central object is the $D_8$-equivariant linear layer, implemented in the Fourier (isotypical) basis of the dihedral group of order 8, whose eight elements are 90-degree rotations and reflections of a square. In that basis any equivariant map between features of type $\frac{C}{8}\rho_{\mathrm{iso}}$ acts independently on each irreducible-representation type, which is what produces the sparse block-diagonal structure and the $5.33\times$ FLOP reduction. The paper carries the same Fourier-domain construction through the whole Transformer block: GELU is applied in the spatial regular representation, attention logits are invariant because the orthogonal feature representations preserve inner products, and the patch embedding and positional encoding are constrained to be $D_8$-equivariant. The hybrid families H8 and I8 set $k=l/2$ to balance the computational saving against the accuracy loss seen in full $D_8$ equivariance.

What would settle it

Train H8(ViT-L/16) with the same DeiT III recipe on a second large classification benchmark where 90-degree rotations are not a natural symmetry, such as text documents, non-oriented microscopy images, or aerial imagery, and compare with the ordinary ViT baseline; if the accuracy gap widens more than the baseline's own transfer drop while FLOP savings stay fixed, the $D_8$ prior is not transferable and the 'match baseline at 40% fewer FLOPs' claim is specific to ImageNet-like data.

Watch

Extended reading notes

Core claim

By enforcing equivariance to the dihedral group $D_8$ in the Fourier (isotypical) domain, equivariant linear layers become block-diagonal intertwiners: Schur's lemma forbids mixing between different irreducible-representation types, so a full $C\times C$ weight matrix collapses to four $C/8\times C/8$ blocks and one $C/4\times C/4$ block. This gives 8x fewer parameters and $16/3 \approx 5.33\times$ fewer FLOPs per linear layer, and the savings carry over to whole ViT blocks as the embedding dimension grows. The empirical discovery is that a hybrid design—octic blocks for the first $k=l/2$ layers, standard blocks after, and a power-spectrum invariantization for the class token—matches or beats the DeiT III and DINOv2 baselines on ImageNet-1K at roughly 61% of the FLOPs, and that the invariant I8 model holds its accuracy on randomly rotated inputs. The fully equivariant $D_8(\mathrm{ViT\text{-}H/14})$ model reaches 82.0% versus the 84.6% baseline, so breaking equivariance in later layers is what preserves accuracy while keeping most of the savings.

Load-bearing premise

The load-bearing premise is that a $D_8$-equivariant feature representation retains the information a ViT needs for ImageNet classification when applied to the first half of the blocks; the paper validates this on ImageNet-1K and a small set of transfer tasks, but the fully equivariant $D_8$ model already loses 2.6 points, so the margin for the hybrid design is real but narrow.

Editorial extensions

If this is right

  • Octic equivariant linear layers give 5.33x fewer FLOPs and up to 8x less memory per feature dimension, and whole-block savings approach this asymptote as embedding dimension grows.
  • H8(ViT-H/14) reaches 85.0% ImageNet top-1 accuracy with 61% of baseline FLOPs, and I8(ViT-H/14) reaches 84.7%, so the accuracy-efficiency trade-off holds for supervised training.
  • The invariant I8 model keeps its accuracy on randomly rotated validation images, where ordinary and non-invariant models degrade by several points.
  • Under the DINOv2 self-supervised recipe, octic ViTs match or slightly improve linear-probe and k-NN ImageNet accuracy, segmentation mIoU on ADE20K and VOC2012, and transfer to Places365 and iNaturalist at lower FLOPs.
  • Breaking equivariance in the later layers is necessary: fully equivariant D8(ViT-H/14) loses about 2.6 accuracy points, while H8 and I8 retain accuracy.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural next test is whether the $D_8$ prior transfers to datasets whose objects lack a canonical orientation; the paper's single hematology transfer result is encouraging, but broader evidence would separate a genuine symmetry prior from ImageNet-specific statistics.
  • The arithmetic-intensity analysis implies octic layers should beat ordinary layers in throughput once the embedding dimension passes roughly 3200; the paper's own benchmarks do not reach that regime, so a direct large-model measurement would separate implementation overhead from the theoretical gain.
  • The $k=l/2$ breakpoint is a design choice validated on ImageNet; testing whether the optimal breakpoint depends on model width, depth, or dataset would determine how safely these layers can be dropped into arbitrary ViT training pipelines.
  • Because the equivariant layers impose a structured block-diagonal sparsity, combining them with activation sparsity or weight quantization could compound the savings without any further architectural change.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces octic (D8) equivariant linear layers for Vision Transformers, implemented in the Fourier domain of the group, and proposes two hybrid ViT families (H8 and I8) that use octic blocks in the first k blocks and standard or invariantized blocks afterward. The central claims are that equivariant linear layers reduce FLOPs by a factor of 16/3 ≈ 5.33 and parameter memory by a factor of 8 relative to ordinary linear layers, and that hybrid models trained with DeiT III and DINOv2 on ImageNet-1K match or exceed baseline accuracy while saving about 40% of FLOPs. The paper also contributes ablations on the number of equivariant blocks and on six invariantization methods.

Significance. If the claims hold, this is a practically useful result: it shows that a classical group-theoretic construction, D8 equivariance, can be turned into a scalable efficiency tool for modern ViTs rather than an added computational burden. The strengths of the paper are the clean and correct complexity accounting in Section 3.3.2, the explicit use of Schur's lemma to derive the 5.33x/8x figures, the breadth of the empirical study (supervised and self-supervised training, transfer probes, several ablations), and the authors' willingness to report actual throughput and memory numbers, including the fact that FLOP savings do not translate one-to-one into wall-clock savings at the sizes tested. The main weakness is empirical: the headline 'match baseline performance' statement rests on comparisons to externally reported baselines and single runs without seed variance, so the parity claim is not yet established to the standard that the abstract and conclusion require.

major comments (3)
  1. [Section 4.1, Table 2, Section E.1] The headline claim that H8 and I8 models 'match or exceed baseline accuracy' is not supported by a controlled comparison. The DeiT III baselines in Table 2 are taken from Touvron et al. (2022) and Bökman et al. (2025), and Section E.1 explicitly states that the authors only train the octic ViTs and compare to figures reported in the respective papers. Moreover, each reported configuration is a single run with no random seeds or variance estimates. The observed differences (+0.4, +0.1, +0.3, -0.2 points for the H8/I8 models at ViT-H/14 and ViT-L/16) are within the run-to-run variation commonly seen in 400-epoch ImageNet training, so they do not by themselves establish parity or superiority. The DINOv2 experiments do train their own baselines, but they are also single runs and are evaluated via linear/k-NN probes rather than the end-to-end supervised setting of the central claim. Since 'match baseline performance' is the load-bearing empirical premise of the abstract and conclusion, the DeiT III baselines need to be retrained under identical conditions, or multiple seeds need to be reported, before this claim can be accepted.
  2. [Section 4.3, Figure 4b] The choice k=l/2 for the number of octic blocks is central to the hybrid models, but its selection protocol is not specified. Figure 4b plots accuracy as a function of k, and the text states that 'k=l/2 strikes a good balance,' which suggests that this value was selected after inspecting results on the ImageNet validation set. Because the fully equivariant model D8(ViT-H/14) loses 2.6 points and D8(ViT-L/16) loses 4.7 points relative to the reported baselines (Table 2), the margin available to the hybrid design is narrow, and selecting k on the same data used to report the headline accuracy risks overfitting the hyperparameter to the validation set. Please state explicitly whether k was fixed a priori, provide validation curves for all k values, or report results on an independent holdout to rule out selection bias.
  3. [Section 4.3, 'Impact of equivariance'] The ablation that supports the claim 'equivariance yields higher accuracy than arbitrary mappings' compares a single-run H8(ViT-B) result of 83.0 with a single-run ablated model of 82.4. This 0.6-point difference has no error bars and is within typical run-to-run noise for this training setup, so it does not by itself demonstrate that the equivariance constraint is the cause of the improvement. Please add repeated seeds or report the variance; if that is not feasible, soften the claim to a qualitative observation.
minor comments (5)
  1. [Abstract / Section 3.3.2] The abstract says 'up to 8x reductions in memory,' but Table 1 reports end-to-end peak-memory improvements of 2.44x–5.80x, and the 8x figure refers to the parameter count of a single linear layer. Please disambiguate 'memory' as weight/parameter memory versus activation/peak memory throughout.
  2. [Section E.1, Table 2] The OOD rotation column is computed on publicly available weights trained for 800 epochs, while the accuracy comparison in the same table uses figures reported for 400 epochs. This makes the OOD delta not directly comparable to the main accuracy numbers; please clarify or recompute on the 400-epoch models.
  3. [Appendix D, Equation (14)] The power spectrum invariantization in Equation (14) uses notation xE1 and xE2, but the earlier notation in Section 3.1.1 defines xE11, xE12, xE21, xE22. Please define xE1 and xE2 or use the earlier notation consistently.
  4. [Table 1] The throughput improvements in Table 1 are described as measured scaling of octic ViTs, but the largest models are not trained. Please clarify in the caption or text whether these throughput numbers are measured on untrained architectural variants and whether they use the same batch size and precision as the standard ViT counterparts.
  5. [Section 4.2, Table 3] The segmentation evaluation for DINOv2 uses the protocol of Darcet et al. (2025), not the original DINOv2 protocol. This is stated in Section E.2, but it would help to note in the main text that segmentation numbers are not directly comparable to the original DINOv2 paper.

Circularity Check

1 steps flagged · score 2.0 of 10

No load-bearing circularity: the FLOP/memory savings are derived from Schur's lemma and explicit block-diagonal counts, and the accuracy claims rest on external or self-trained baselines; the only by-construction 'finding' is the I8 rotation-invariance check, which does not support the headline efficiency claim.

  1. self definitional [Section 4.1, paragraph immediately after Table 2]
    "In the final column of Table 2, we study the effect of evaluating models on a randomly rotated validation set without training on such augmentations. We find that the invariant model performs equally well while the performance of the remaining models (including H8) significantly degrade."

    The I8 architecture is defined (Section 3.2) to map D8-steerable features to D8-invariant A1 features before classification, and Eq. (10) makes the full pre-invariantization feature map D8-equivariant. Therefore the top-1 prediction of I8 is exactly invariant to 90-degree rotations by construction; reporting that it 'performs equally well' on a randomly rotated validation set is a consistency check entailed by the definition, not an empirical discovery. The empirical content is only the degradation of the non-invariant models. This step is not load-bearing for the paper's central FLOP/accuracy claims, so it only mildly raises the circularity score.

full rationale

The central efficiency derivation is self-contained: Section 3.3.2 counts the equivariant linear layer's parameters and FLOPs directly from the block-diagonal isotypical decomposition, and the 16/3 FLOP ratio and 8x parameter reduction follow from Schur's lemma, not from any fitted quantity. The accuracy-matching claim is supported by external DeiT III baselines (Touvron et al., 2022) and by self-trained DINOv2 baselines, so it is not a relabeled fit. The self-citation to Bökman et al. (2025) supplies the Fourier-domain technique, but the technique is re-derived here from representation theory and is not the sole justification for the efficiency result. The only by-construction statement identified is the I8 rotation-invariance check described above; it is a sanity check consistent with the architecture and does not carry the paper's main conclusions. Concerns about single-run baseline comparisons and external baselines are statistical-correctness issues, not circularity.

Assumptions & free parameters 2 free parameters · 5 assumptions · 0 invented entities

The paper leans on standard representation theory rather than invented entities. The hand-chosen quantities are architectural: the equivariant block fraction k=l/2 and the power-spectrum invariantization, both selected from ablations. These affect the headline results but are not hidden constants in a derivation.

free parameters (2)
  • k (number of octic blocks) = l/2 (12 for ViT-L, 16 for ViT-H)
    Chosen from the ablation in Figure 4b; the headline H8/I8 accuracy-efficiency results use this split and would change with k.
  • invariantization function = power spectrum with K=6
    Selected from six candidates in Appendix D, Table 5; I8 and D8 results depend on this choice.
assumptions (5)
  • standard math Schur's lemma and the isotypic decomposition of D8 representations.
    Used in Section 3.1 and 3.3.2 to justify that equivariant linear layers are block-diagonal with no mixing between non-isomorphic irreps.
  • standard math The regular representation of D8 decomposes as A1⊕A2⊕B1⊕B2⊕2E, with Qreg given in Appendix A.
    Example 3.3 and equation (11); the efficient implementation uses this explicit Fourier basis.
  • domain assumption Natural image classification benefits from a D8 equivariance prior in early layers.
    Introduction and Figure 1; this is tested empirically, not proven, and full equivariance actually hurts accuracy.
  • domain assumption The DeiT III and DINOv2 training recipes transfer to octic architectures without re-tuning.
    Section 4 and the Limitations section; hyperparameters are taken from prior work unchanged.
  • domain assumption The power-spectrum invariantization retains enough information for classification after k octic blocks.
    Appendix D, Table 5; chosen over five alternatives based on validation accuracy.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Quick ViTs: Speeding up Vision Transformers through Equivariance." pith.science (2026). https://pith.science/paper/IJ4D7AKK

@misc{pith2026250515441,
  author       = {Pith},
  title        = {Pith review of: Quick ViTs: Speeding up Vision Transformers through Equivariance},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/IJ4D7AKK}},
  note         = {Machine review of arXiv:2505.15441}
}
abstract

Natural images exhibit strong geometric regularities: local structures, such as edges, corners, and textures, appear in many orientations and mirror configurations. Since Vision Transformers (ViTs) operate on square image patches, these transformations naturally correspond to the dihedral symmetry group $\mathrm{D}_8$, also known as the octic group. Recent work has shown that ViTs can be made reflection equivariant and more efficient than standard ViTs simultaneously by implementing the linear layers in the Fourier domain of the reflection group. In this work, we extend the equivariance to reflections and rotations and analyze the scalability of the resulting networks. Our Quick ViTs, based on octic equivariant linear layers, achieve 5.33x reductions in FLOPs and up to 8x reductions in memory compared to ordinary linear layers. By analyzing the arithmetic intensity of these layers, we identify theoretical limits on how much the FLOP savings translate into throughput improvements on modern GPUs. However, these limitations disappear as the embedding dimensions increase. Enabled by their computational efficiency, we conduct a broader empirical evaluation of equivariant ViTs than in previous work. Upon training supervised (DeiT-III) and self-supervised (DINOv2) on ImageNet-1K, we find that our Quick ViTs match or exceed baseline accuracy while at the same time providing substantial efficiency gains.

Figures

Figures reproduced from arXiv: 2505.15441 by the authors.

Figure 1
Figure 1. Computational savings. Using octic layers in ViTs significantly reduces the computational complexity without sacrificing accuracy on ImageNet-1K, for both supervised and self-supervised training. Detailed results can be found in Section 4. In this paper, we demonstrate that scaling equivariance to larger groups can be efficiently implemented – yielding faster, stronger, and more compact models, cf [PITH_FULL_IMAGE:… view at source ↗
Figure 2
Figure 2. D8 Linear layers. Implementing equivariant linear layers in the Fourier domain of D8 gives a major computational benefit. Left: A C × C weight matrix being multiplied by L tokens of feature dimension C. Center: The block-diagonalization that happens when enforcing the layer to be D8-equivariant in the Fourier domain. More precisely, we enforce equivariance with respect to the representation C 8 ρiso that splits into… view at source ↗
Figure 3
Figure 3. Architecture. Patches are first extracted from an image using specialized octic filters and the resulting features are processed by k octic ViT blocks. The final embeddings can be fed to l − k standard Transformer blocks (as demonstrated by our H8 and I8 ViTs). When k = l, we denote I8(ViT) by D8(ViT), which hence only uses octic ViT blocks before a final invariantization. 3.3 OCTIC LAYERS In this section we detail … view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: (a) Reduction in FLOPs from a non-equivariant Transformer block to an octic-equivariant block vs. embedding dimension. The matmul ratio reflects only matrix multiplications in linear layers and Attention; the total ratio includes all computations. (b) The effect of cha…
Figure 5
Figure 5. Figure 5: (a) PatchEmbed filters from a trained network. More filters are shown in [PITH_FULL_IMAGE:figures/full_fig_p017_5.png]
Figure 6
Figure 6. Figure 6: Comparison of learned patch embedding filters. (a) DeiT III. (b) DINOv2. Each figure shows the top-16 principal components of the octic PatchEmbed filter for a specific feature type. 18 [PITH_FULL_IMAGE:figures/full_fig_p018_6.png]
Figure 7
Figure 7. Figure 7: DINOv2 training progression. Classification accuracy development during 125K training steps for linear probe and k-NN on frozen features for ViT-L sized models. 21 [PITH_FULL_IMAGE:figures/full_fig_p021_7.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Platonic Transformers: A Solid Choice For Equivariance

    cs.CV 2025-10 conditional novelty 6.0 of 10

    Platonic Transformers achieve exact equivariance to translations plus discrete Platonic-solid rotations by lifting features into multiple reference frames and sharing one RoPE attention across them, with a linear-time...

Reference graph

Works this paper leans on

69 extracted references · 42 canonical work pages · cited by 1 Pith paper

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should not add it explicitly Type <Return> for now, but then later remove the command n...

  3. [3]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@first@sw \@firstoftwo \@ifundefined NAT@b*@#2 \@firstoftwo @num @NAT@ctr \@secondoft...

  4. [4]

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibsetup #1 @NAT@ctr @ @openbib .11em \@plus.33em \@minus.07em 4000 4000 `\.\@m @bibit...

    work page

  5. [5]

    Accurate structure prediction of biomolecular interactions with alphafold 3

    Josh Abramson, Jonas Adler, Jack Dunger, Richard Evans, Tim Green, Alexander Pritzel, Olaf Ronneberger, Lindsay Willmore, Andrew J Ballard, Joshua Bambrick, et al. Accurate structure prediction of biomolecular interactions with alphafold 3. Nature, 630 0 (8016): 0 493--500, 2024

    2024

  6. [6]

    Getting vit in shape: Scaling laws for compute-optimal model design

    Ibrahim M Alabdulmohsin, Xiaohua Zhai, Alexander Kolesnikov, and Lucas Beyer. Getting vit in shape: Scaling laws for compute-optimal model design. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neural Information Processing Systems, volume 36, pp.\ 16406--16425. Curran Associates, Inc., 2023. URL https://proceed...

    work page 2023

  7. [7]

    Vn-transformer: Rotation-equivariant attention for vector neurons

    Serge Assaad, Carlton Downey, Rami Al-Rfou, Nigamaa Nayakanti, and Ben Sapp. Vn-transformer: Rotation-equivariant attention for vector neurons. Transactions on Machine Learning Research, 1 2023

    work page 2023

  8. [8]

    How to scale your model

    Jacob Austin, Sholto Douglas, Roy Frostig, Anselm Levskaya, Charlie Chen, Sharad Vikram, Federico Lebron, Peter Choy, Vinay Ramasesh, Albert Webson, and Reiner Pope. How to scale your model. Online, 2025. Retrieved from https://jax-ml.github.io/scaling-book/

    work page 2025

Show all 69 references
  1. [9]

    Roto-translation covariant convolutional networks for medical image analysis

    Erik J Bekkers, Maxime W Lafarge, Mitko Veta, Koen AJ Eppenhof, Josien PW Pluim, and Remco Duits. Roto-translation covariant convolutional networks for medical image analysis. In Medical Image Computing and Computer Assisted Intervention--MICCAI 2018: 21st International Confer...

    2018

  2. [10]

    Erik J Bekkers, Sharvaree Vadgama, Rob Hesselink, Putri A Van der Linden, and David W. Romero. Fast, expressive \ mathrm\ SE \ (n)\ equivariant networks through weight-sharing in position-orientation space. In The Twelfth International Conference on Learning Representations, 2...

    2024

  3. [11]

    An Efficient Sparse Kernel Generator for O(3)-Equivariant Deep Networks

    Vivek Bharadwaj, Austin Glover, Aydin Buluc, and James Demmel. An Efficient Sparse Kernel Generator for O(3)-Equivariant Deep Networks. Society for Industrial and Applied Mathematics, 2025. URL https://arxiv.org/abs/2501.13986

    2025 arXiv

  4. [12]

    o kman, David Nordstr \

    Georg B \"o kman, David Nordstr \"o m, and Fredrik Kahl. Flopping for flops: Leveraging equivariance for computational efficiency. In Forty-second International Conference on Machine Learning, 2025

    2025

  5. [13]

    Does equivariance matter at scale? Transactions on Machine Learning Research, 2025

    Johann Brehmer, S \"o nke Behrends, Pim De Haan, and Taco Cohen. Does equivariance matter at scale? Transactions on Machine Learning Research, 2025. ISSN 2835-8856. URL https://openreview.net/forum?id=wilNute8Tn

    2025

  6. [14]

    Group-invariant max filtering

    Jameson Cahill, Joseph W Iverson, Dustin G Mixon, and Daniel Packer. Group-invariant max filtering. Foundations of Computational Mathematics, pp.\ 1--38, 2024

    2024

  7. [15]

    End-to-end object detection with transformers

    Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In Computer Vision – ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part I, pp.\ 213...

    2020 doi

  8. [16]

    Sparsevit: Revisiting activation sparsity for efficient high-resolution vision transformer

    Xuanyao Chen, Zhijian Liu, Haotian Tang, Li Yi, Hang Zhao, and Song Han. Sparsevit: Revisiting activation sparsity for efficient high-resolution vision transformer. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023

    2023

  9. [17]

    Group equivariant convolutional networks

    Taco Cohen and Max Welling. Group equivariant convolutional networks. In ICML, 2016

    2016

  10. [18]

    Steerable CNN s

    Taco Cohen and Max Welling. Steerable CNN s. In ICLR, 2017

    2017

  11. [19]

    Flash A ttention-2: Faster attention with better parallelism and work partitioning

    Tri Dao. Flash A ttention-2: Faster attention with better parallelism and work partitioning. In International Conference on Learning Representations (ICLR), 2024

    2024

  12. [20]

    Cluster and predict latents patches for improved masked image modeling

    Timothée Darcet, Federico Baldassarre, Maxime Oquab, Julien Mairal, and Piotr Bojanowski. Cluster and predict latents patches for improved masked image modeling. Transactions on Machine Learning Research, feb 2025. Published February 12, 2025

    2025

  13. [21]

    Mostafa Dehghani, Josip Djolonga, Basil Mustafa, Piotr Padlewski, Jonathan Heek, Justin Gilmer, Andreas Peter Steiner, Mathilde Caron, Robert Geirhos, Ibrahim Alabdulmohsin, Rodolphe Jenatton, Lucas Beyer, Michael Tschannen, Anurag Arnab, Xiao Wang, Carlos Riquelme Ruiz, Matth...

    2023

  14. [22]

    Imagenet: A large-scale hierarchical image database

    Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp.\ 248--255, 2009. doi:10.1109/CVPR.2009.5206848

    2009

  15. [23]

    Exploiting cyclic symmetry in convolutional neural networks

    Sander Dieleman, Jeffrey De Fauw, and Koray Kavukcuoglu. Exploiting cyclic symmetry in convolutional neural networks. In International conference on machine learning, pp.\ 1889--1898. PMLR, 2016

    2016

  16. [24]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at...

    2021

  17. [25]

    RoMa: Robust Dense Feature Matching

    Johan Edstedt, Qiyu Sun, Georg Bökman, Mårten Wadenbäck, and Michael Felsberg. RoMa: Robust Dense Feature Matching . IEEE Conference on Computer Vision and Pattern Recognition, 2024

    2024

  18. [26]

    The pascal visual object classes (voc) challenge

    Mark Everingham, Luc Van Gool, Christopher Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. International Journal of Computer Vision, 88: 0 303--338, 06 2010. doi:10.1007/s11263-009-0275-4

    2010 doi

  19. [27]

    The invariantring package for macaulay2

    Luigi Ferraro, Federico Galetto, Francesca Gandini, Hang Huang, Matthew Mastroeni, and Xianglong Ni. The invariantring package for macaulay2. Journal of Software for Algebra and Geometry, 14 0 (1): 0 5--11, 2024

    2024

  20. [28]

    Fuchs, Daniel E

    Fabian B. Fuchs, Daniel E. Worrall, Volker Fischer, and Max Welling. Se(3)-transformers: 3d roto-translation equivariant attention networks. In Advances in Neural Information Processing Systems 34 (NeurIPS), 2020

    2020

  21. [29]

    Grayson and Michael E

    Daniel R. Grayson and Michael E. Stillman. Macaulay2, a software system for research in algebraic geometry. Available at http://www2.macaulay2.com

  22. [30]

    Neighborhood attention transformer

    Ali Hassani, Steven Walton, Jiachen Li, Shen Li, and Humphrey Shi. Neighborhood attention transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp.\ 6185--6194, June 2023

    2023

  23. [31]

    Efficient equivariant network

    Lingshen He, Yuxuan Chen, zhengyang shen, Yiming Dong, Yisen Wang, and Zhouchen Lin. Efficient equivariant network. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, volume 34, pp.\ 5290--53...

    2021

  24. [32]

    Gaussian error linear units (gelus)

    Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). arXiv preprint arXiv:1606.08415, 2016

    2016 arXiv

  25. [33]

    Lietransformer: Equivariant self-attention for lie groups

    Michael J Hutchinson, Charline Le Lan, Sheheryar Zaidi, Emilien Dupont, Yee Whye Teh, and Hyunjik Kim. Lietransformer: Equivariant self-attention for lie groups. In Proceedings of the 38th International Conference on Machine Learning (ICML), pp.\ 4533--4543. PMLR, 2021

    2021

  26. [34]

    Equivariance with learned canonicalization functions

    S\' e kou-Oumar Kaba, Arnab Kumar Mondal, Yan Zhang, Yoshua Bengio, and Siamak Ravanbakhsh. Equivariance with learned canonicalization functions. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), Proceedings of th...

    2023

  27. [35]

    The bispectrum as a source of phase-sensitive invariants for fourier descriptors: a group-theoretic approach

    Ramakrishna Kakarala. The bispectrum as a source of phase-sensitive invariants for fourier descriptors: a group-theoretic approach. Journal of Mathematical Imaging and Vision, 44: 0 341--353, 2012

    2012

  28. [36]

    Scaling laws for neural language models

    Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, 2020

    2001 arXiv

  29. [37]

    Berg, Wan-Yen Lo, Piotr Dollar, and Ross Girshick

    Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo, Piotr Dollar, and Ross Girshick. Segment anything. In Proceedings of the IEEE/CVF International Conference on Computer Vision ...

    2023

  30. [38]

    Dinobloom: a foundation model for generalizable cell embeddings in hematology

    Valentin Koch, Sophia J Wagner, Salome Kazeminia, Ece Sancar, Matthias Hehr, Julia A Schnabel, Tingying Peng, and Carsten Marr. Dinobloom: a foundation model for generalizable cell embeddings in hematology. In International Conference on Medical Image Computing and Computer-As...

    2024

  31. [39]

    Steerable transformers for volumetric data

    Soumyabrata Kundu and Risi Kondor. Steerable transformers for volumetric data. In Forty-second International Conference on Machine Learning, 2025

    2025

  32. [40]

    Equiformer: Equivariant graph attention transformer for 3d atomistic graphs

    Yi-Lun Liao and Tess Smidt. Equiformer: Equivariant graph attention transformer for 3d atomistic graphs. In International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=KwmPfARgOTD

    2023

  33. [41]

    Swin transformer: Hierarchical vision transformer using shifted windows

    Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pp.\ 9992--10002, 2021. doi:10.1109/ICCV489...

    2021

  34. [42]

    An expert-annotated dataset of bone marrow cytology in hematologic malignancies

    Christian Matek, Sebastian Krappe, Christian Münzenmayer, Torsten Haferlach, and Carsten Marr. An expert-annotated dataset of bone marrow cytology in hematologic malignancies. Data set, 2021. URL https://doi.org/10.7937/TCIA.AXH3-T579

    2021 doi

  35. [43]

    cuEquivariance : High-performance equivariant neural networks

    NVIDIA. cuEquivariance : High-performance equivariant neural networks. URL https://docs.nvidia.com/cuda/cuequivariance/index.html

  36. [44]

    Maxime Oquab, Timoth \'e e Darcet, Th \'e o Moutakanni, Huy V. Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel HAZIZA, Francisco Massa, Alaaeldin El-Nouby, Mido Assran, Nicolas Ballas, Wojciech Galuba, Russell Howes, Po-Yao Huang, Shang-Wen Li, Ishan Misra, Micha...

    2024

  37. [45]

    Pytorch: An imperative style, high-performance deep learning library

    Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu ...

    2019

  38. [46]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In International Con...

    2021

  39. [47]

    Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do I mage N et classifiers generalize to I mage N et? In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of...

    2019

  40. [48]

    Rojas-Gomez, Teck-Yian Lim, Minh N

    Renan A. Rojas-Gomez, Teck-Yian Lim, Minh N. Do, and Raymond A. Yeh. Making vision transformers truly shift-equivariant. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp.\ 5568--5577, June 2024

    2024

  41. [49]

    Attentive group equivariant convolutional networks

    David Romero, Erik Bekkers, Jakub Tomczak, and Mark Hoogendoorn. Attentive group equivariant convolutional networks. In International Conference on Machine Learning, pp.\ 8188--8199. PMLR, 2020

    2020

  42. [50]

    Imagenet large scale visual recognition challenge

    Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115: 0 211--252, 2015

    2015

  43. [51]

    A general framework for robust g-invariance in g-equivariant networks

    Sophia Sanborn and Nina Miolane. A general framework for robust g-invariance in g-equivariant networks. Advances in Neural Information Processing Systems, 36: 0 67103--67124, 2023

    2023

  44. [52]

    Linear Representations of Finite Groups , volume 42 of Graduate Texts in Mathematics

    Jean-Pierre Serre. Linear Representations of Finite Groups , volume 42 of Graduate Texts in Mathematics . Springer, New York, NY, 1977. ISBN 978-1-4684-9460-0 978-1-4684-9458-7. doi:10.1007/978-1-4684-9458-7. URL http://link.springer.com/10.1007/978-1-4684-9458-7

    1977 doi

  45. [53]

    Philippe Tillet, H. T. Kung, and David Cox. Triton: an intermediate language and compiler for tiled neural network computations. In Proceedings of the 3rd ACM SIGPLAN International Workshop on Machine Learning and Programming Languages, MAPL 2019, pp.\ 10–19, New York, NY, USA...

    2019

  46. [54]

    Deit iii: Revenge of the vit

    Hugo Touvron, Matthieu Cord, and Herv \'e J \'e gou. Deit iii: Revenge of the vit. In Shai Avidan, Gabriel Brostow, Moustapha Ciss \'e , Giovanni Maria Farinella, and Tal Hassner (eds.), Computer Vision -- ECCV 2022, pp.\ 516--533, Cham, 2022. Springer Nature Switzerland. ISBN...

    2022

  47. [55]

    On the utility of equivariance and symmetry breaking in deep learning architectures on point clouds

    Sharvaree Vadgama, Mohammad Mohaiminul Islam, Domas Buracus, Christian Shewmake, and Erik Bekkers. On the utility of equivariance and symmetry breaking in deep learning architectures on point clouds. arXiv preprint arXiv:2501.01999, 2025

    2025 arXiv

  48. [56]

    Benchmarking representation learning for natural world image collections

    Grant Van Horn, Elijah Cole, Sara Beery, Kimberly Wilber, Serge Belongie, and Oisin Mac Aodha. Benchmarking representation learning for natural world image collections. In Computer Vision and Pattern Recognition, 2021

    2021

  49. [57]

    Attention is all you need

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Info...

    2017

  50. [58]

    Vggt: Visual geometry grounded transformer, 2025

    Jianyuan Wang, Minghao Chen, Nikita Karaev, Andrea Vedaldi, Christian Rupprecht, and David Novotny. Vggt: Visual geometry grounded transformer, 2025. URL https://arxiv.org/abs/2503.11651

    2025 arXiv

  51. [59]

    Dust3r: Geometric 3d vision made easy

    Shuzhe Wang, Vincent Leroy, Yohann Cabon, Boris Chidlovskii, and Jerome Revaud. Dust3r: Geometric 3d vision made easy. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp.\ 20697--20709, June 2024 a

    2024

  52. [60]

    Swallowing the bitter pill: Simplified scalable conformer generation

    Yuyang Wang, Ahmed AA Elhag, Navdeep Jaitly, Joshua M Susskind, and Miguel \'A ngel Bautista. Swallowing the bitter pill: Simplified scalable conformer generation. In International Conference on Machine Learning, pp.\ 50400--50418. PMLR, 2024 b

    2024

  53. [61]

    General E(2) -equivariant steerable CNN s

    Maurice Weiler and Gabriele Cesa. General E(2) -equivariant steerable CNN s. In NeurIPS, 2019. URL https://proceedings.neurips.cc/paper/2019/file/45d6637b718d0f24a237069fe41b0db4-Paper.pdf

    2019

  54. [62]

    Pytorch image models

    Ross Wightman. Pytorch image models. https://github.com/rwightman/pytorch-image-models, 2019

    2019

  55. [63]

    Representation theory and invariant neural networks

    Jeffrey Wood and John Shawe-Taylor . Representation theory and invariant neural networks. Discrete Applied Mathematics, 69 0 (1-2): 0 33--60, August 1996. ISSN 0166218X. doi:10/c3qmr6

    1996

  56. [64]

    Cvt: Introducing convolutions to vision transformers

    Haiping Wu, Bin Xiao, Noel Codella, Mengchen Liu, Xiyang Dai, Lu Yuan, and Lei Zhang. Cvt: Introducing convolutions to vision transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp.\ 22--31, October 2021

    2021

  57. [65]

    e (2) -equivariant vision transformer

    Renjun Xu, Kaifan Yang, Ke Liu, and Fengxiang He. e (2) -equivariant vision transformer. In Uncertainty in Artificial Intelligence, pp.\ 2356--2366. PMLR, 2023

    2023

  58. [66]

    Scaling vision transformers

    Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer. Scaling vision transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp.\ 12104--12113, June 2022

    2022

  59. [67]

    Places: A 10 million image database for scene recognition

    Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: A 10 million image database for scene recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017 a

    2017

  60. [68]

    Scene parsing through ade20k dataset

    Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017 b

    2017

  61. [69]

    Semantic understanding of scenes through the ade20k dataset

    Bolei Zhou, Hang Zhao, Xavier Puig, Tete Xiao, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Semantic understanding of scenes through the ade20k dataset. International Journal of Computer Vision, 127 0 (3): 0 302--321, 2019

    2019

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.