REVIEW 21 cited by
ImageNet-21K Pretraining for the Masses
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
ImageNet-21K Pretraining for the Masses
read the original abstract
ImageNet-1K serves as the primary dataset for pretraining deep learning models for computer vision tasks. ImageNet-21K dataset, which is bigger and more diverse, is used less frequently for pretraining, mainly due to its complexity, low accessibility, and underestimation of its added value. This paper aims to close this gap, and make high-quality efficient pretraining on ImageNet-21K available for everyone. Via a dedicated preprocessing stage, utilization of WordNet hierarchical structure, and a novel training scheme called semantic softmax, we show that various models significantly benefit from ImageNet-21K pretraining on numerous datasets and tasks, including small mobile-oriented models. We also show that we outperform previous ImageNet-21K pretraining schemes for prominent new models like ViT and Mixer. Our proposed pretraining pipeline is efficient, accessible, and leads to SoTA reproducible results, from a publicly available dataset. The training code and pretrained models are available at: https://github.com/Alibaba-MIIL/ImageNet21K
Forward citations
Cited by 21 Pith papers
-
CanViT: Toward Active-Vision Foundation Models
CanViT is the first task- and policy-agnostic AVFM pretrained via passive-to-active dense latent distillation on 13.2M scenes and 1B random glimpses, achieving 38.5% ADE20K mIoU in one glimpse and 84.5% ImageNet-1k to...
-
Condensing Large-Scale Datasets Directly with Minimal Information Loss
CIM directly aligns data distributions to condense large-scale datasets with minimal information loss, achieving new SOTA results on ImageNet-1K distillation at IPC=10.
-
Identifying Latent Concepts and Structures for Generalized Category Discovery
CPF-GCD enforces low-rank compositional structure on vision backbone features via spatial primitive fields so that novel categories emerge as new activation patterns over a shared vocabulary of reusable visual primitives.
-
The Edge-on Galaxies in the DESI survey (EGIDE): sample building and photometry
The EGIDE project releases a tenfold larger catalogue of edge-on galaxies with griz photometry, stellar masses, redshifts and star formation rates, finding that red-sequence galaxies are thicker than blue-cloud ones a...
-
ExDet: Open-Domain Open-Vocabulary Detection with Cross-modal Extrapolation and Rectification
ExDet proposes a lightweight framework using text-guided extrapolation, detector-compatible rectification, and recalibrated proposals to achieve state-of-the-art open-domain open-vocabulary detection on multiple benchmarks.
-
VISReg: Variance-Invariance-Sketching Regularization for JEPA training
VISReg replaces covariance in VICReg-style objectives with sliced-Wasserstein sketching for JEPA training, claiming better OOD performance and resilience to collapse.
-
LV-OSD: Language-Vision-Complementary Open-Set Object Detection
The authors define the LV-OSD problem and introduce the LVDor dual-branch framework with TPDW dynamic weighting and PRM masking to align multimodal prompts for open-set detection.
-
Energy-Structured Low-Rank Adaptation for Continual Learning
E²-LoRA structures low-rank adaptations by energy concentration and ordering with dynamic rank allocation to achieve state-of-the-art continual learning.
-
Weierstrass Positional Encoding for Vision Transformers
WePE encodes 2D patch positions in Vision Transformers via Weierstrass elliptic functions on the complex plane to exploit double periodicity and derive relative positions algebraically.
-
Birds of a Feather Flock Together: Background-Invariant Representations via Linear Structure in VLMs
Exploiting linear structure in VLM embeddings, a synthetic-data pre-training method yields background-invariant representations that exceed 90% worst-group accuracy on Waterbirds even under 100% spurious correlation w...
-
Parameter-Efficient Adaptation of Pre-Trained Vision Foundation Models for Active and Passive Seismic Data Denoising
Adapting vision foundation models with LoRA and kurtosis-guided unsupervised test-time adaptation matches or exceeds domain-specific models for seismic denoising across multiple sites and unseen data.
-
Contrastive Semantic Projection: Faithful Neuron Labeling with Contrastive Examples
Using contrastive examples with vision-language models and a new CLIP-based scoring method called CSP produces more faithful and granular neuron labels than prior activation-only approaches.
-
StableTTA: Improving Vision Model Performance by Training-free Test-Time Adaptation Methods
StableTTA improves ImageNet-1K accuracy across 71 vision models by stabilizing logit aggregation under coherent-batch inference and enabling efficient single-forward-pass adaptation.
-
MePo: Meta Post-Refinement for Rehearsal-Free General Continual Learning
MePo refines pretrained backbones via meta-learning on constructed pseudo tasks and initializes a meta covariance matrix to enable robust second-order alignment, yielding 12-15% gains on CIFAR-100, ImageNet-R and CUB-...
-
LanguageBind: Extending Video-Language Pretraining to N-modality by Language-based Semantic Alignment
LanguageBind aligns video, infrared, depth, and audio to a frozen language encoder via contrastive learning on the new VIDAL-10M dataset, extending video-language pretraining to N modalities.
-
JetViT: Efficient High-Resolution Vision Transformer with Post-Training Attention Search
JetViT uses post-training attention search to hybridize full-attention ViTs with linear and window attention blocks, achieving up to 1.79x throughput gains on high-res images while preserving accuracy on DINOv3 and De...
-
MobileVLM : A Fast, Strong and Open Vision Language Assistant for Mobile Devices
MobileVLM achieves on-par performance with much larger vision-language models on standard benchmarks while delivering state-of-the-art inference speeds of 21.5 tokens per second on Snapdragon 888 CPU and 65.3 on Jetso...
-
Page image classifier fine-tuned on century-spanning archives of scanned documents for further content-specific processing
Fine-tuned RegNetY-16GF reaches 99.16% accuracy classifying 48k century-old Czech archival pages into 11 visual content types, beating a 75% hand-crafted feature baseline, with models and data released publicly.
-
HandyLabel: Towards Post-Processing to Real-Time Annotation Using Skeleton Based Hand Gesture Recognition
HandyLabel enables real-time data annotation by mapping hand gestures to labels, with ResNet50 on skeleton-preprocessed HaGRID data reaching 0.923 F1-score and 88.9% of 46 study participants preferring it to tradition...
-
Page image classification for content-specific data processing
Develops an image classification system for historical document pages to support content-specific analysis pipelines.
-
4th Workshop on Maritime Computer Vision (MaCVi): Challenge Overview
The report overviews five maritime computer vision benchmark challenges, their datasets, protocols, quantitative results, and top team approaches from the MaCVi 2026 workshop.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.