REVIEW 4 cited by
Getting ViT in Shape: Scaling Laws for Compute-Optimal Model Design
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Getting ViT in Shape: Scaling Laws for Compute-Optimal Model Design
read the original abstract
Scaling laws have been recently employed to derive compute-optimal model size (number of parameters) for a given compute duration. We advance and refine such methods to infer compute-optimal model shapes, such as width and depth, and successfully implement this in vision transformers. Our shape-optimized vision transformer, SoViT, achieves results competitive with models that exceed twice its size, despite being pre-trained with an equivalent amount of compute. For example, SoViT-400m/14 achieves 90.3% fine-tuning accuracy on ILSRCV2012, surpassing the much larger ViT-g/14 and approaching ViT-G/14 under identical settings, with also less than half the inference cost. We conduct a thorough evaluation across multiple tasks, such as image classification, captioning, VQA and zero-shot transfer, demonstrating the effectiveness of our model across a broad range of domains and identifying limitations. Overall, our findings challenge the prevailing approach of blindly scaling up vision models and pave a path for a more informed scaling.
Forward citations
Cited by 4 Pith papers
-
RADIO1D: Elastic Representations for Condensed Vision Modeling
RADIO1D produces elastic hierarchical 1D visual tokens via multi-teacher distillation that match or beat fixed 2D encoders in VLMs at lower token counts.
-
Scaling Laws for Neural-Network Quantum States
Transformer wave functions for the J1-J2 Heisenberg model exhibit size-independent power-law decay of V-score with compute, with the exponent decreasing as frustration increases.
-
Image Recognition with Vision and Language Embeddings of VLMs
A benchmark of dual-encoder VLMs finds text and image embeddings give complementary class accuracy, and a per-class precision fusion rule adds about 0.4% accuracy over either alone on ImageNet.
-
Vector embedding of multi-modal texts: a tool for discovery?
Using ColPali embeddings of 3,600 textbook page images, cosine similarity beats dot product, Euclidean, and Manhattan distances on top-5 retrieval, but only reaches 0.51 precision@5 without a text-only baseline.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.