Pith. sign in

REVIEW 4 major objections 4 minor 14 references

Enhancing compact convolutional transformers with super attention

T0 review · 4 major / 4 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that replacing scaled dot-product attention with a token-mixing 'super attention' in a compact convolutional transformer improves CIFAR-100 top-1 accuracy from 36.50% to 46.29% while cutting parameters to 10.6M from 17.7M.

desk verdict A compact transformer variant that swaps SDPA for Hosseini et al.'s token mixing reports a 10-point CIFAR100 gain against a self-implemented baseline that scores 36.5%; the claim is not supported until the baseline is fixed. read the letter →

arxiv 2508.18960 v1 pith:3NBNQ35Y submitted 2025-08-26 cs.CV cs.LG

Simpenzwe Honore Leandre , Natenaile Asmamaw Shiferaw , Dillip Rout This is my paper
classification cs.CVcs.LG
keywords compactconvolutionaltransformersuperattentiontokenmixingscaleddotproductCIFAR-100sequencepoolingvisionparameterefficiency
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that the standard scaled dot-product attention (SDPA) used in vision transformers contains redundant value projections, and that replacing it with a token-mixing 'super attention' block yields both higher accuracy and lower parameter count on fixed-context-length vision tasks. Using a compact convolutional transformer backbone with a convolutional tokenizer and sequence pooling, the authors report a jump from 36.50% to 46.29% top-1 validation accuracy on CIFAR-100 (66.33% to 76.31% top-5) in 75 epochs, while total parameters drop from 17.7M to 10.6M. They further claim these gains come without any data augmentation, positional embeddings, or learning-rate scheduling, and that super attention is more efficient than SDPA when the token sequence is shorter than the embedding dimension. If true, this would show that attention modules can be made substantially simpler and cheaper for small-scale vision tasks without sacrificing accuracy.

What carries the argument

Super attention, the paper's name for token mixing: instead of computing each head's values with a learned linear projection V_i = X W_i^V, it applies a single learned sequence-mixing matrix W^A ∈ R^{ℓ×ℓ} to the value vectors from the left, multiplying across the token dimension. This mixes information across the ℓ tokens and eliminates per-head value projections. The block also uses compact convolutional transformer pieces: a one-layer 3×3 convolutional tokenizer that embeds patches, and sequence pooling (softmax-weighted pooling over the sequence) in place of a class token. The mechanism's claimed effect is to reduce attention parameters by 25% and total parameters by 40% while producing m

What would settle it

Train the same CCT-6/3x1 backbone with a properly tuned SDPA attention module (standard augmentation, learning-rate schedule, original CCT hyperparameters) on CIFAR-100 for 75 epochs. If that baseline reaches or exceeds 46.29% top-1 validation accuracy, the claimed advantage of super attention over SDPA is falsified.

Watch

Extended reading notes

Core claim

On CIFAR-100, a six-layer compact convolutional transformer with super attention reaches 46.29% top-1 and 76.31% top-5 validation accuracy, versus 36.50% and 66.33% for the same backbone with standard SDPA, within the same 75-epoch budget. The model uses 10.6M parameters, about 60% of the 17.7M SDPA baseline; attention parameters drop by roughly 25% and total parameters by 40%. The authors attribute the gain to token mixing: a learned ℓ×ℓ matrix applied to the value vectors from the left, replacing per-head value projections in SDPA. They also report that the SDPA baseline overfits and diverges in validation loss while super attention converges stably, without augmentation, positional embedd

Load-bearing premise

The SDPA baseline is a correctly implemented, reasonably tuned compact transformer, so the 10-point accuracy gain comes from replacing attention with token mixing rather than from an undertrained or misconfigured baseline.

Editorial extensions

If this is right

  • If the result holds, token-mixing attention is a drop-in replacement for SDPA in compact transformers, improving CIFAR-100 accuracy by roughly 10 points at 60% of the parameter count.
  • Super attention becomes the preferred attention when context length is smaller than embedding dimension (e.g., 32×32 images with 768-dim embeddings), since its cost is lower than SDPA's in that regime.
  • Training stability without mixup, positional embeddings, or a learning-rate schedule suggests the model could be easier to deploy in low-resource settings.
  • The same convolutional tokenizer plus sequence pooling plus token-mixing design may transfer to other fixed-length recognition benchmarks beyond CIFAR-100, as the authors expect.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural next experiment is to re-run the SDPA baseline with the original CCT hyperparameters and standard augmentation to see how much of the 10-point gap is recovered; the paper's 36.50% baseline sits far below typical published CCT-6/3x1 numbers on CIFAR-100, so the size of the true token-mixing gain is not yet pinned down.
  • If the efficiency analysis carries over, token-mixing attention could lower the inference cost of small vision transformers on edge hardware, since it avoids per-head value projections and has favorable complexity when tokens are few.
  • The authors' removal of positional embeddings aligns with earlier work showing convolutional tokenizers can encode locality; an ablation that swaps the tokenizer for plain patching would separate the tokenizer's contribution from the token-mixing block's.
  • Scaling the model up (more layers, larger datasets like ImageNet) would test whether the stability and accuracy gains persist beyond the fixed 75-epoch CIFAR-100 regime.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The manuscript proposes a variant of the Compact Convolutional Transformer (CCT) in which the standard scaled dot-product attention (SDPA) is replaced by 'super attention' (token mixing with a learned l x l matrix, following Hosseini et al. 2025), combined with sequence pooling and a convolutional tokenizer. On CIFAR-100, the authors report top-1/top-5 validation accuracies of 46.29%/76.31% for their model versus 36.50%/66.33% for a self-implemented CCT-6/3x1 with SDPA, while reducing parameters from 17.7M to 10.6M. Training uses 75 epochs, constant AdamW lr=0.01, batch size 1024, no data augmentation, and no learning-rate schedule. The paper claims state-of-the-art performance, high training stability, and efficiency gains when context length is below embedding dimension.

Significance. If correct, the result would be an interesting data point: token-mixing attention could beat SDPA in a low-budget CCT setting with 40% fewer parameters and without augmentation or LR schedules. The manuscript is transparent about its limitations and provides explicit equations and hyperparameters, which is useful. The main claim, however, is not established by the current evidence: the comparison uses a single seed, a baseline that is far outside the normal operating range of CCT, and no code or error bars. The efficiency claim is also not measured. The practical significance is therefore conditional on a properly controlled comparison.

major comments (4)
  1. [§4, Table 2; Figure 4] The central 10-point gain rests on an unvalidated SDPA baseline. CCT-6/3x1 implemented by the authors reaches only 36.50% top-1 on CIFAR-100, far below typical published CCT-6/3x1 results (often >75% under standard training recipes). The training configuration in Table 1 (constant lr=0.01, batch 1024, no augmentation, no schedule, 75 epochs) is atypical; Figure 4 shows the SDPA model's validation loss increasing while training loss drops, i.e., severe overfitting. The comparison therefore conflates the attention mechanism with training robustness: super attention may simply tolerate the degenerate schedule better. Please report (a) a properly tuned SDPA baseline using standard CCT training practices, (b) published CCT-6/3x1 numbers in the same table, and (c) multi-seed mean±std for both models.
  2. [Abstract; §1; §4] The abstract and introduction describe the result as 'state-of-the-art performance', but Table 2 contains no state-of-the-art comparison: only a ViT-12/16 entry from a different training setup and the authors' own low-scoring baseline. The title and abstract overstate the contribution. Either add a comparison table with published compact transformer results on CIFAR-100, or remove the SOTA wording.
  3. [§4, Table 1] The evidence consists of a single run. No seeds, error bars, or code URL appear in the manuscript despite the statement 'We make our code available on Github.' Single-run comparisons cannot exclude initialization or run-to-run variance, especially with lr=0.01 constant. Provide the code repository and at least three seeds.
  4. [Abstract; §4] The efficiency claim ('more efficient ... when context length is less than embedding dimension') is not substantiated. Parameter count is reported (10.6M vs 17.7M), but no latency, memory, FLOPs, or throughput measurements are given. The theoretical complexity tradeoff should be stated explicitly and, if possible, measured on the T4 used for training.
minor comments (4)
  1. [Figures 1-2; throughout] 'Top 1%' and 'top 5%' should be 'top-1' and 'top-5' accuracy; the current phrasing is misleading.
  2. [Figure 4] The text says the baseline 'diverges', but the curves show overfitting: training loss decreases while validation loss increases. Please rephrase.
  3. [§3.1] The notation for the token-mixing matrix W^A is ambiguous: is W^A shared across heads or per-head? Equations (2)-(5) suggest per-head, but the description says W^A ∈ R^{l×l} without a head subscript.
  4. [§3.2] Typo: 'CIF AR100' should be 'CIFAR-100'. Also mention the dataset resolution consistently.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity; the paper reports an empirical comparison on held-out CIFAR-100 and does not derive its claimed outcomes from fitted parameters or self-citations.

full rationale

The paper's central claim—that its 'super attention' variant improves CIFAR-100 top-1 accuracy from 36.50% to 46.29% relative to its own SDPA baseline—is an empirical result evaluated on a held-out validation split. The method is assembled from previously published components (token mixing from Hosseini et al. 2025, sequence pooling and convolutional tokenizers from Hassani et al. 2022), but these are external prior contributions, not self-citations by the present authors, and the paper does not claim to derive the accuracy gains mathematically. The attention equations in Section 3.1 are standard and are not defined in terms of the reported accuracies; no fitted parameter is renamed as a prediction, and no self-referential uniqueness theorem is invoked. The underperforming SDPA baseline (36.50% top-1 versus much higher published CCT results) is a serious experimental-validity concern, but it is a matter of benchmark fairness and correctness risk, not circularity. Likewise, the unsupported 'state-of-the-art' wording and the claim that 'we expect similar performance regardless' are overclaims, but they do not make the derivation circular. The paper is self-contained in the sense that its reported comparisons are direct empirical measurements, so the appropriate circularity score is 0.

Assumptions & free parameters 7 free parameters · 5 assumptions · 1 invented entities

The architecture combines components from two cited prior works (Hosseini et al. 2025, Hassani et al. 2022) plus a DeepSeek-V3 style block. All hyperparameters are hand-picked with no search or sensitivity analysis. The central claim depends on the assumption that the SDPA baseline is properly configured, which is not supported by the reported low accuracy.

free parameters (7)
  • embedding dimension (embed_dim) = 768
    Hand-chosen; central to parameter count and context-length scaling claims. No sensitivity analysis.
  • number of heads = 24
    Hand-chosen; interacts with token mixing's value mixing matrix and head count.
  • number of transformer layers = 6
    Hand-chosen depth; the paper claims 6 layers but does not vary this.
  • learning rate = 0.01 (constant)
    Constant LR, no schedule; this is known to be problematic for many transformers, so it may contribute to the weak baseline.
  • batch size = 1024
    Hand-chosen; affects training stability and generalization.
  • dropout rate = 0.3
    Hand-chosen; affects overfitting behavior that the paper attributes to the attention mechanism.
  • weight decay = 0.01
    Hand-chosen; part of AdamW setup.
assumptions (5)
  • domain assumption Token mixing equations (1)-(5) as defined by Hosseini et al. (2025) are valid and implemented as specified.
    The core attention replacement is taken verbatim from a cited paper; no re-derivation or ablation is provided.
  • domain assumption Sequence pooling (equations 6-8) and the convolutional tokenizer from Hassani et al. (2022) compose correctly with token mixing.
    The paper combines them but does not test each component separately.
  • domain assumption The DeepSeek-V3 transformer block, with no positional embeddings, is a sound base for both models.
    Adapted from DeepSeek-AI et al. (2025); the paper does not analyze normalization or MLP details.
  • domain assumption CIFAR100 results generalize to other vision tasks.
    The paper only tests CIFAR100 and even acknowledges this limitation.
  • ad hoc to paper Constant learning rate with AdamW is a fair hyperparameter choice for the SDPA baseline.
    The baseline may be disadvantaged by the same constant LR; no hyperparameter search is reported.
invented entities (1)
  • Super attention
    purpose: Named replacement for the multi-head SDPA block in the proposed CCT variant
    Equations (1)-(5) are identical to token mixing in Hosseini et al. (2025); 'super attention' is a new brand, not a new mechanism, and there is no falsifiable handle specific to the rename.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Enhancing compact convolutional transformers with super attention." pith.science (2026). https://pith.science/paper/3NBNQ35Y

@misc{pith2026250818960,
  author       = {Pith},
  title        = {Pith review of: Enhancing compact convolutional transformers with super attention},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/3NBNQ35Y}},
  note         = {Machine review of arXiv:2508.18960}
}
read the original abstract

In this paper, we propose a vision model that adopts token mixing, sequence-pooling, and convolutional tokenizers to achieve state-of-the-art performance and efficient inference in fixed context-length tasks. In the CIFAR100 benchmark, our model significantly improves the baseline of the top 1% and top 5% validation accuracy from 36.50% to 46.29% and 66.33% to 76.31%, while being more efficient than the Scaled Dot Product Attention (SDPA) transformers when the context length is less than the embedding dimension and only 60% the size. In addition, the architecture demonstrates high training stability and does not rely on techniques such as data augmentation like mixup, positional embeddings, or learning rate scheduling. We make our code available on Github.

Figures

Figures reproduced from arXiv: 2508.18960 by the authors.

Figure 1
Figure 1. Top 1% Training and validation accuracy comparison between standard atten [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Top 5% Training and validation accuracy comparison between standard atten [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. Diagram of transformer backbone. Linear* is transformation from the left. [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Training and validation loss comparison between standard attention (SDPA) and [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

14 extracted references · 3 canonical work pages

  1. [4]

    Adi Haviv, Ori Ram, Ofir Press, Peter Izsak, and Omer Levy

    URL https: //arxiv.org/abs/2104.05704. Adi Haviv, Ori Ram, Ofir Press, Peter Izsak, and Omer Levy. Transformer language mod- els without positional encodings still learn positional information. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang, editors, Findings of the Association for Compu- tational Linguistics: EMNLP 2022, pages 1382–1390, Abu Dhabi, U...

    arXiv 2022

  2. [5]

    doi: 10.18653/v1/ 2022.findings-emnlp.99

    Association for Computational Linguistics. doi: 10.18653/v1/ 2022.findings-emnlp.99. URL https://aclanthology.org/2022.findings-emnlp.99/. Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus),

    doi:10.18653/v1/ 2022

  3. [7]

    Cost-Effective Attention Mechanisms for Low Resource Settings: Necessity & Sufficiency of Linear Transformations

    URL https://arxiv.org/abs/2403.01643. Wei Hu, Lechao Xiao, and Jeffrey Pennington. Provable benefit of orthogonal initialization in optimizing deep linear networks. In International Conference on Learning Represen- tations,

    work page Pith review arXiv

  4. [10]

    doi: https://doi.org/10.1016/j.neucom.2023.127063

    ISSN 0925-2312. doi: https://doi.org/10.1016/j.neucom.2023.127063. URL https: //www.sciencedirect.com/science/article/pii/S0925231223011864. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, L ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, ...

    arXiv 2023

  5. [12]

    Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo

    URL https://arxiv.org/ abs/2103.15808. Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy to train strong classifiers with lo- calizable features,

    arXiv

  6. [2014]

    Jay Shah, Ganesh Bikshandi, Ying Zhang, Vijay Thakkar, Pradeep Ramani, and Tri Dao

    URL https://arxiv.org/ abs/1312.6120. Jay Shah, Ganesh Bikshandi, Ying Zhang, Vijay Thakkar, Pradeep Ramani, and Tri Dao. Flashattention-3: Fast and accurate attention with asynchrony and low-precision,

    arXiv

  7. [2017]

    Haiping Wu, Bin Xiao, Noel Codella, Mengchen Liu, Xiyang Dai, Lu Yuan, and Lei Zhang

    URL https://proceedings.neurips.cc/paper_files/paper/2017/ file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf. Haiping Wu, Bin Xiao, Noel Codella, Mengchen Liu, Xiyang Dai, Lu Yuan, and Lei Zhang. Cvt: Introducing convolutions to vision transformers,

    work page 2017

  8. [2018]

    URL https://arxiv.org/abs/1710.09412. 9

    arXiv

Show all 14 references
  1. [2019]

    Hongyi Zhang, Moustapha Cisse, Yann N

    URL https://arxiv.org/abs/1905.04899. Hongyi Zhang, Moustapha Cisse, Yann N. Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization,

    1905 arXiv

  2. [2021]

    Ali Hassani, Steven Walton, Nikhil Shah, Abulikemu Abuduweili, Jiachen Li, and Humphrey Shi

    URL https://arxiv.org/abs/2010.11929. Ali Hassani, Steven Walton, Nikhil Shah, Abulikemu Abuduweili, Jiachen Li, and Humphrey Shi. Escaping the big data paradigm with compact transformers,

    2010 arXiv

  3. [2022]

    URL https://proceedings.neurips.cc/paper_files/paper/2022/file/ 67d57c32e20fd0a7a302cb81d36e40d5-Paper-Conference.pdf. DeepSeek-AI, Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Daya Guo,...

    2022

  4. [2023]

    Peyman Hosseini, Mehran Hosseini, Ignacio Castro, and Matthew Purver

    URL https: //arxiv.org/abs/1606.08415. Peyman Hosseini, Mehran Hosseini, Ignacio Castro, and Matthew Purver. Cost-effective attention mechanisms for low resource settings: Necessity & sufficiency of linear trans- formations,

    arXiv

  5. [2024]

    Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhut- dinov

    URL https://arxiv.org/abs/2407.08608. Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhut- dinov. Dropout: A simple way to prevent neural networks from overfitting. Journal of 8 Machine Learning Research, 15(56):1929–1958,

    1929 arXiv

  6. [2025]

    URL https://arxiv.org/abs/2412.19437. Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words:...

    arXiv

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.