Pith. sign in

REVIEW 25 cited by

Are we done with ImageNet?

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2006.07159 v1 pith:FWD32HDP submitted 2020-06-12 cs.CV cs.LG

Are we done with ImageNet?

Lucas Beyer , Olivier J. H\'enaff , Alexander Kolesnikov , Xiaohua Zhai , A\"aron van den Oord This is my paper
classification cs.CV cs.LG
keywords imagenetlabelsfindoriginalprocedurebenchmarkwhetheraccuracy
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved
0 comments
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Yes, and no. We ask whether recent progress on the ImageNet classification benchmark continues to represent meaningful generalization, or whether the community has started to overfit to the idiosyncrasies of its labeling procedure. We therefore develop a significantly more robust procedure for collecting human annotations of the ImageNet validation set. Using these new labels, we reassess the accuracy of recently proposed ImageNet classifiers, and find their gains to be substantially smaller than those reported on the original labels. Furthermore, we find the original ImageNet labels to no longer be the best predictors of this independently-collected set, indicating that their usefulness in evaluating vision models may be nearing an end. Nevertheless, we find our annotation procedure to have largely remedied the errors in the original labels, reinforcing ImageNet as a powerful benchmark for future research in visual recognition.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 25 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. CanViT: Toward Active-Vision Foundation Models

    cs.CV 2026-03 conditional novelty 8.0

    CanViT is the first task- and policy-agnostic AVFM pretrained via passive-to-active dense latent distillation on 13.2M scenes and 1B random glimpses, achieving 38.5% ADE20K mIoU in one glimpse and 84.5% ImageNet-1k to...

  2. LAION-C: An Out-of-Distribution Benchmark for Web-Scale Vision Models

    cs.CV 2025-06 accept novelty 8.0

    LAION-C supplies six novel corruptions that stay OOD for web-scale training sets and demonstrates that leading models now rival or exceed human robustness on them.

  3. From Uncertain Judgments to Calibrated Rankings: Conformal Elo Estimation for LLM Evaluation

    cs.LG 2026-06 unverdicted novelty 7.0

    SoftElo uses calibrated win probabilities in Bradley-Terry models plus split conformal prediction on held-out residuals to produce LLM-derived Elo ratings within 17.9 MAE of human ratings along with distribution-free ...

  4. SCOOTER: A Human Evaluation Framework for Unrestricted Adversarial Examples

    cs.CV 2025-07 conditional novelty 7.0

    SCOOTER supplies best-practice guidelines, open tools, and a 3K-image benchmark with 34K+ human ratings showing that six tested unrestricted attacks produce images humans can detect as fake.

  5. Wake Vision: A Tailored Dataset and Benchmark Suite for TinyML Computer Vision Applications

    cs.CV 2024-05 unverdicted novelty 7.0

    Wake Vision pipeline produces a 6M-image person detection dataset for TinyML with 2.2% label error, improving model accuracy up to 6.6% over prior VWW benchmark across architectures and subsets.

  6. PaLI: A Jointly-Scaled Multilingual Language-Image Model

    cs.CV 2022-09 conditional novelty 7.0

    PaLI jointly scales a 4B-parameter vision transformer with language models on a new 10B multilingual image-text dataset to reach state-of-the-art results on vision-language tasks while keeping a simple modular design.

  7. Sharpness-Aware Minimization and Muon: Robustness under the Spectral Norm

    cs.LG 2026-07 conditional novelty 6.0

    A spectral-norm inner perturbation combined with a Muon outer update achieves the best ImageNet validation accuracy among the compared SAM variants on both a ViT-Small/16 and a ResNet-50.

  8. Consistent Evidence, Robust Recognition: Faithful Attribution Regularization under Geometric Transformations

    cs.CV 2026-07 conditional novelty 6.0

    Submodular region-search attributions plus a ranking/truncation loss regularize models toward spatially corresponding evidence under geometric transforms, improving attribution metrics and transformed accuracy with sm...

  9. Evaluating Epistemic Uncertainty: Beyond OOD Detection and Active Learning

    cs.LG 2026-07 conditional novelty 6.0

    Epistemic uncertainty should be judged by how well it ranks reducible error, and a new Pareto-gap diagnostic shows proxy-task rankings can invert.

  10. Unmasking LAION-5B: Age, Gender, Race, and Emotion Biases in Large-Scale Image Datasets

    cs.CV 2026-06 unverdicted novelty 6.0

    Empirical audit of LAION-2B-en and LAION-2B-multi finds overrepresentation of young adults, White people, and males plus stereotypical emotion associations across two attribute classifiers.

  11. Elastic Attention Cores for Scalable Vision Transformers

    cs.CV 2026-05 unverdicted novelty 6.0

    VECA learns effective visual representations using core-periphery attention where patches interact exclusively via a resolution-invariant set of learned core embeddings, achieving linear O(N) complexity while maintain...

  12. Franca: Nested Matryoshka Clustering for Scalable Visual Representation Learning

    cs.CV 2025-07 conditional novelty 6.0

    Franca introduces nested Matryoshka clustering and positional disentanglement in a transparent SSL pipeline to deliver open-source vision models competitive with closed proprietary systems.

  13. Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling

    cs.CV 2024-12 unverdicted novelty 6.0

    InternVL 2.5 is the first open-source MLLM to surpass 70% on the MMMU benchmark via model, data, and test-time scaling, with a 3.7-point gain from chain-of-thought reasoning.

  14. Vision Transformers Need Registers

    cs.CV 2023-09 unverdicted novelty 6.0

    Adding register tokens to Vision Transformers eliminates high-norm background artifacts and raises state-of-the-art performance on dense visual prediction tasks.

  15. Sigmoid Loss for Language Image Pre-Training

    cs.CV 2023-03 conditional novelty 6.0

    SigLIP replaces softmax-based contrastive loss with a simple pairwise sigmoid loss for vision-language pre-training, decoupling batch size from normalization and reaching strong zero-shot performance with limited compute.

  16. Mixture-of-Control: State-Aware Fine-Tuning for Transformer-based Models

    cs.LG 2026-06 unverdicted novelty 5.0

    Mixture-of-Control adaptively combines local and global control states in transformer fine-tuning by treating per-block states as experts in a sparse MoE setup to improve cross-block communication while keeping memory...

  17. Detecting Regional Spurious Correlations in Vision Transformers via Token Discarding

    cs.CV 2025-09 conditional novelty 5.0

    A token-discarding method for vision transformers measures whether predictions rely on features outside the object's bounding box, identifying spurious correlations and problematic ImageNet classes.

  18. InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks

    cs.CV 2023-12 unverdicted novelty 5.0

    InternVL scales a vision model to 6B parameters and aligns it with LLMs using web data to achieve state-of-the-art results on 32 visual-linguistic benchmarks.

  19. DINOv2: Learning Robust Visual Features without Supervision

    cs.CV 2023-04 unverdicted novelty 5.0

    Pith review generated a malformed one-line summary.

  20. Riemannian Deep Learning: Modules, Networks, and Geometries

    cs.LG 2026-07 accept novelty 4.0

    One Lie-group/gyrogroup framework unifies batch normalization and logistic-regression classifiers across SPD, rotation, correlation, Grassmannian, and constant-curvature manifolds, with new hyperbolic and SPD geometries.

  21. Position: Early-Stage Quality Assurance in Annotation Pipelines Is More Cost-Effective Than Late-Stage Validation

    cs.SE 2026-05 unverdicted novelty 4.0

    Early-stage QA in annotation pipelines is more cost-effective than late-stage validation, supported by a proposed taxonomy of trigger points and a parametric error-propagation model.

  22. Hierarchical Pre-Training of Vision Encoders with Large Language Model

    cs.CV 2026-03 reject novelty 4.0

    A three-stage pre-training scheme that feeds multi-layer vision features into an LLM reports marginal benchmark gains, but lacks data, code, and ablations needed to support the claim.

  23. Image Recognition with Vision and Language Embeddings of VLMs

    cs.CV 2025-09 conditional novelty 4.0

    A benchmark of dual-encoder VLMs finds text and image embeddings give complementary class accuracy, and a per-class precision fusion rule adds about 0.4% accuracy over either alone on ImageNet.

  24. SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Features

    cs.CV 2025-02 unverdicted novelty 4.0

    SigLIP 2 models trained with a unified recipe of captioning, self-supervised losses, and curated diverse data outperform prior SigLIP versions on classification, retrieval, localization, dense prediction, and multilin...

  25. PaLI-X: On Scaling up a Multilingual Vision and Language Model

    cs.CV 2023-05 unverdicted novelty 4.0

    Scaling a multilingual vision-language model in size and training breadth yields new state-of-the-art results on over 25 benchmarks plus emerging abilities in counting and multilingual detection.