Pith. sign in

REVIEW 6 cited by

An Image is Worth More Than 16x16 Patches: Exploring Transformers on Individual Pixels

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2406.09415 v2 pith:JKAGFNVO submitted 2024-06-13 cs.CV cs.LG

Duy-Kien Nguyen , Mahmoud Assran , Unnat Jain , Martin R. Oswald , Cees G. M. Snoek , Xinlei Chen This is my paper · ORCID
classification cs.CVcs.LG
keywords visioncomputerindividualarchitecturesbiasdirectlygenerationimage
verification ladder T0 review T1 audit T2 compute T3 formal

Signed reviews

No signed human review yet.

0 comments
read the original abstract

This work does not introduce a new method. Instead, we present an interesting finding that questions the necessity of the inductive bias of locality in modern computer vision architectures. Concretely, we find that vanilla Transformers can operate by directly treating each individual pixel as a token and achieve highly performant results. This is substantially different from the popular design in Vision Transformer, which maintains the inductive bias from ConvNets towards local neighborhoods (e.g. by treating each 16x16 patch as a token). We showcase the effectiveness of pixels-as-tokens across three well-studied computer vision tasks: supervised learning for classification and regression, self-supervised learning via masked autoencoding, and image generation with diffusion models. Although it's computationally less practical to directly operate on individual pixels, we believe the community must be made aware of this surprising piece of knowledge when devising the next generation of neural network architectures for computer vision.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 6 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Hierarchical Compact Clustering Attention (COCA) for Unsupervised Object-Centric Learning

    cs.CV 2025-05 conditional novelty 6.0 of 10

    COCA-Net introduces compactness-guided hierarchical clustering within an attention architecture, achieving state-of-the-art unsupervised object segmentation on synthetic multi-object images.

  2. Fast Vision Mamba: Pooling Spatial Dimensions for Accelerated Processing

    cs.CV 2025-02 conditional novelty 5.0 of 10

    FastVim reduces Vision Mamba's SSM parallel scan steps from log(h^2) to log(h) by alternately mean-pooling tokens across rows or columns, delivering up to a 72.5% inference speedup at 2048x2048 with roughly unchanged ...

  3. A Study on Context Length and Efficient Transformers for Biomedical Image Analysis

    cs.CV 2024-12 conditional novelty 5.0 of 10

    Smaller image patches improve transformer accuracy on biomedical tasks, attention-window size matters less, and Hyena/MambaVision operators match attention with up to 80% faster training.

  4. LLaVA-NeuMT: Selective Layer-Neuron Modulation for Efficient Multilingual Multimodal Translation

    cs.CL 2025-07 conditional novelty 4.0 of 10

    Selectively fine-tuning only 40% of a multimodal LLM's layers and neurons can match or slightly beat full fine-tuning on multilingual image-to-text translation benchmarks, though the measured gains are marginal.

  5. Dynamic Token Selection for Aerial-Ground Person Re-Identification

    cs.CV 2024-11 reject novelty 4.0 of 10

    A top-K token selector added to a view-decoupled transformer improves aerial-ground person re-identification accuracy on CARGO and AG-ReID, but the paper's specifications are internally inconsistent.

  6. A Decade of Deep Learning: A Survey on The Magnificent Seven

    cs.LG 2024-12 reject novelty 2.0 of 10

    This is a review of seven influential deep learning models that does not present new research results and suffers from methodological and integrity issues.

Pith tools