REVIEW 10 cited by
Patches Are All You Need?
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Patches Are All You Need?
read the original abstract
Although convolutional networks have been the dominant architecture for vision tasks for many years, recent experiments have shown that Transformer-based models, most notably the Vision Transformer (ViT), may exceed their performance in some settings. However, due to the quadratic runtime of the self-attention layers in Transformers, ViTs require the use of patch embeddings, which group together small regions of the image into single input features, in order to be applied to larger image sizes. This raises a question: Is the performance of ViTs due to the inherently-more-powerful Transformer architecture, or is it at least partly due to using patches as the input representation? In this paper, we present some evidence for the latter: specifically, we propose the ConvMixer, an extremely simple model that is similar in spirit to the ViT and the even-more-basic MLP-Mixer in that it operates directly on patches as input, separates the mixing of spatial and channel dimensions, and maintains equal size and resolution throughout the network. In contrast, however, the ConvMixer uses only standard convolutions to achieve the mixing steps. Despite its simplicity, we show that the ConvMixer outperforms the ViT, MLP-Mixer, and some of their variants for similar parameter counts and data set sizes, in addition to outperforming classical vision models such as the ResNet. Our code is available at https://github.com/locuslab/convmixer.
Forward citations
Cited by 10 Pith papers
-
DECKER: Domain-invariant Embedding for Cross-Keyboard Extraction and Recognition
DECKER improves cross-keyboard and cross-user keystroke identification from audio using domain-invariant embeddings on the new diverse HEAR dataset, with additional gains from language model sequence correction.
-
TCD-Arena: Assessing Robustness of Time Series Causal Discovery Methods Against Assumption Violations
TCD-Arena is a new customizable testing framework that runs millions of experiments to map how 33 different assumption violations affect time series causal discovery methods and shows ensembles can boost overall robustness.
-
UltraViT: Latency-Optimized On-device Vision Encoder for Large Vision-Language Models
A heterogeneous pyramidal vision encoder plus two-stage dense-then-generative pre-training yields faster, stronger on-device LVLM vision encoding than FastVLM.
-
SWIFT: Spatio-temporal Wavelet Integrated Forecasting Framework for Workload Traces
SWIFT forecasts cloud workloads at SOTA accuracy with linear complexity by combining learnable cascaded wavelet convolutions and sequential inter/intra-variable mixing.
-
DECKER: Domain-invariant Embedding for Cross-Keyboard Extraction and Recognition
DECKER is a domain-invariant four-stage framework (keyboard normalization, adversarial disentanglement, cross-keyboard contrastive alignment, acoustic style randomization) plus LLM post-processing that improves keystr...
-
Towards Real-Time ECG and EMG Modeling on $\mu$NPUs
PhysioLite delivers Transformer-comparable ECG/EMG performance using learnable wavelet filters and hardware-aware design at ~370KB quantized size on μNPUs.
-
Transolver: A Fast Transformer Solver for PDEs on General Geometries
Transolver learns intrinsic physical states from discretized meshes by adaptively splitting domains into flexible learnable slices and computing attention over physics-aware tokens, achieving state-of-the-art PDE solv...
-
Learning High-Level Decision Making with an Interaction-Aware Attention-Based Network in Autonomous Driving
An attention architecture that bottlenecks traffic agents into fixed latent queries plus a finer discrete action set yields higher simulated speeds and lower early-termination rates than DeepSet and Ego-attention on t...
-
LALE: Lightweight-Transformer Architecture for Land-Cover Estimation
LALE introduces a bifurcated ConvMixer-transformer encoder with an all-MLP decoder for efficient semantic segmentation of remote sensing imagery, achieving near-baseline F1 scores with 4.5x fewer parameters on the ARA...
-
AI in Agriculture: A Survey of Deep Learning Techniques for Crops, Fisheries and Livestock
A systematic survey of over 200 works on deep learning and AI techniques for crops, fisheries, and livestock in agriculture.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.