pith. sign in

arxiv: 2303.15389 · v1 · submitted 2023-03-27 · 💻 cs.CV

EVA-CLIP: Improved Training Techniques for CLIP at Scale

Quan Sun , Yuxin Fang , Ledell Wu , Xinlong Wang , Yue Cao This is my paper

Pith reviewed 2026-05-13 01:48 UTC · model grok-4.3

classification 💻 cs.CV
keywords CLIPcontrastive learningzero-shot classificationImageNetvision-language modelstraining efficiencyrepresentation learningaugmentation
0
0 comments X

The pith

EVA-CLIP applies new techniques in representation learning, optimization, and augmentation to train CLIP models that reach higher zero-shot ImageNet accuracy with far fewer samples than prior versions.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper develops EVA-CLIP, a family of models that refines contrastive language-image pre-training through targeted changes to representation learning, optimization procedures, and data augmentation strategies. These changes aim to deliver stronger zero-shot performance on image classification tasks while cutting the amount of training data and compute required for a given model size. The largest variant, a 5-billion-parameter model, reaches 82.0 percent zero-shot top-1 accuracy on ImageNet-1K validation after seeing only 9 billion samples. A smaller 430-million-parameter model achieves 80.4 percent accuracy after 6 billion samples. If correct, the work shows that deliberate engineering choices can make large vision-language models more efficient without sacrificing capability.

Core claim

By incorporating new techniques for representation learning, optimization, and augmentation, EVA-CLIP models achieve superior performance relative to previous CLIP models of equivalent parameter count while requiring substantially smaller training costs. The largest model, EVA-02-CLIP-E/14+ with 5.0 billion parameters, attains 82.0 zero-shot top-1 accuracy on ImageNet-1K validation after only 9 billion seen samples. The EVA-02-CLIP-L/14+ model with 430 million parameters reaches 80.4 accuracy after 6 billion samples.

What carries the argument

The suite of new representation learning, optimization, and augmentation techniques applied to contrastive language-image pre-training.

If this is right

  • CLIP models of fixed size can attain higher zero-shot accuracy after seeing fewer total samples.
  • Training compute budgets for high-performing vision-language models can be reduced while preserving accuracy.
  • Smaller models can reach accuracy levels that previously required larger parameter counts or more data.
  • Open release of the trained models lowers the barrier for downstream research on efficient pre-training.
  • The same techniques can be applied when scaling models further without a proportional rise in data needs.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the techniques generalize beyond the reported setups, they could improve training efficiency for other contrastive multimodal objectives.
  • Lower sample requirements might make it feasible to train specialized vision-language models in settings with limited data access.
  • The approach invites testing whether comparable refinements to representation, optimization, or augmentation steps yield gains in non-contrastive self-supervised vision tasks.
  • Success here suggests that systematic tuning of training components can shift the scaling curves for large vision-language models.

Load-bearing premise

The reported performance gains arise chiefly from the new representation learning, optimization, and augmentation techniques rather than from undisclosed differences in training data scale, curation, or hardware.

What would settle it

Train a standard CLIP model on exactly the same data volume and hardware as EVA-CLIP but without the new techniques, then measure whether its zero-shot ImageNet-1K accuracy matches or falls short of the reported 82.0 and 80.4 percent figures.

read the original abstract

Contrastive language-image pre-training, CLIP for short, has gained increasing attention for its potential in various scenarios. In this paper, we propose EVA-CLIP, a series of models that significantly improve the efficiency and effectiveness of CLIP training. Our approach incorporates new techniques for representation learning, optimization, and augmentation, enabling EVA-CLIP to achieve superior performance compared to previous CLIP models with the same number of parameters but significantly smaller training costs. Notably, our largest 5.0B-parameter EVA-02-CLIP-E/14+ with only 9 billion seen samples achieves 82.0 zero-shot top-1 accuracy on ImageNet-1K val. A smaller EVA-02-CLIP-L/14+ with only 430 million parameters and 6 billion seen samples achieves 80.4 zero-shot top-1 accuracy on ImageNet-1K val. To facilitate open access and open research, we release the complete suite of EVA-CLIP to the community at https://github.com/baaivision/EVA/tree/master/EVA-CLIP.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 2 minor

Summary. The paper introduces EVA-CLIP, a family of CLIP-style models that incorporate new techniques for representation learning, optimization, and augmentation. These are claimed to yield superior zero-shot performance on ImageNet-1K at reduced training cost relative to prior CLIP models of comparable parameter count. Concrete results include 82.0% top-1 accuracy for the 5.0B-parameter EVA-02-CLIP-E/14+ model after 9 billion seen samples and 80.4% for the 430M-parameter EVA-02-CLIP-L/14+ after 6 billion seen samples. The full suite of models is released publicly.

Significance. If the reported gains can be attributed to the proposed techniques rather than differences in data curation or scale, the work would constitute a practical advance in efficient large-scale contrastive vision-language pre-training. The public model release supports reproducibility and downstream use.

major comments (2)
  1. [Abstract and §4] Abstract and §4 (Experiments): the central attribution of the reported accuracy gains (e.g., 82.0% and 80.4% zero-shot top-1) to the new representation, optimization, and augmentation techniques is not isolated, because no controlled re-training of prior CLIP baselines on identical data composition, filtering, and sample count is presented.
  2. [§4 and Table 1] §4 and Table 1: no error bars, multiple random seeds, or variance estimates are reported for the zero-shot accuracies despite the known sensitivity of CLIP training to hyperparameters and data order.
minor comments (2)
  1. [§3] §3: the description of the new augmentation pipeline would benefit from an explicit list of all transforms and their probabilities to aid exact reproduction.
  2. [Figure 2] Figure 2: axis labels and legend text are too small for comfortable reading at standard print size.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive feedback on our manuscript. We address each major comment below with clarifications on our experimental design and indicate the revisions we will incorporate to improve the presentation of results and limitations.

read point-by-point responses

  1. Referee: [Abstract and §4] Abstract and §4 (Experiments): the central attribution of the reported accuracy gains (e.g., 82.0% and 80.4% zero-shot top-1) to the new representation, optimization, and augmentation techniques is not isolated, because no controlled re-training of prior CLIP baselines on identical data composition, filtering, and sample count is presented.

    Authors: We agree that fully isolating the contribution of our proposed techniques would require controlled re-training of prior CLIP baselines under identical data composition, filtering, and sample counts. Such experiments are computationally prohibitive at the scale of billions of samples and multiple model sizes. Our comparisons follow standard practice in the field by referencing the best-reported results from the original publications, which used their own data pipelines. The core contribution of EVA-CLIP lies in the combined training recipe that achieves the reported accuracies with substantially fewer seen samples. In the revised manuscript, we will expand the discussion in §4 to explicitly acknowledge potential data differences as a confounding factor and clarify that the efficiency gains are demonstrated relative to published baselines under our unified recipe. revision: partial

  2. Referee: [§4 and Table 1] §4 and Table 1: no error bars, multiple random seeds, or variance estimates are reported for the zero-shot accuracies despite the known sensitivity of CLIP training to hyperparameters and data order.

    Authors: We recognize the value of variance estimates given the sensitivity of CLIP training to hyperparameters and data ordering. However, the computational cost of training models at the scales presented (up to 5B parameters over billions of samples) made multiple independent runs with different random seeds infeasible within our resource constraints. We performed single runs per configuration, consistent with reporting practices in comparable large-scale pre-training works. In the revised version, we will add a dedicated paragraph in §4 discussing this limitation, the known sensitivity of contrastive training, and the consistency of improvements across model scales as supporting evidence for the reliability of the trends. revision: partial

Circularity Check

0 steps flagged

No circularity detected in derivation or claims

full rationale

The paper describes empirical training techniques for CLIP models (representation learning, optimization, augmentation) and reports zero-shot accuracies on the external ImageNet-1K benchmark. No equations, derivations, or predictions are presented that reduce by construction to fitted inputs or self-referential definitions. Central performance claims are validated against public external data rather than internal self-citations or renamed patterns. The work is self-contained against external benchmarks with no load-bearing self-citation chains or ansatz smuggling.

Axiom & Free-Parameter Ledger

1 free parameters · 1 axioms · 0 invented entities

The work rests on the standard CLIP contrastive objective and common vision transformer architectures; no new entities are postulated.

free parameters (1)
  • training hyperparameters
    Batch size, learning rate schedules, and augmentation strengths are tuned but not enumerated in the abstract.
axioms (1)
  • standard math Contrastive image-text matching loss
    Core objective inherited from original CLIP without modification.

pith-pipeline@v0.9.0 · 5492 in / 1020 out tokens · 42753 ms · 2026-05-13T01:48:55.298479+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Lean theorems connected to this paper

Citations machine-checked in the Pith Canon. Every link opens the source theorem in the public Lean library.

What do these tags mean?
matches
The paper's claim is directly supported by a theorem in the formal canon.
supports
The theorem supports part of the paper's argument, but the paper may add assumptions or extra steps.
extends
The paper goes beyond the formal theorem; the theorem is a base layer rather than the whole result.
uses
The paper appears to rely on the theorem as machinery.
contradicts
The paper's claim conflicts with a theorem or certificate in the canon.
unclear
Pith found a possible connection, but the passage is too broad, indirect, or ambiguous to say the theorem truly supports the claim.

Forward citations

Cited by 60 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. MolSight: Molecular Property Prediction with Images

    cs.CV 2026-05 unverdicted novelty 8.0

    Vision encoders on single 2D molecular images with a chemistry-informed curriculum achieve top or near-top results on 10 property prediction tasks at 80x lower FLOPs than multi-modal competitors.

  2. Challenging Vision-Language Models with Physically Deployable Multimodal Semantic Lighting Attacks

    cs.CV 2026-04 unverdicted novelty 8.0

    MSLA is the first physically deployable attack that uses adversarial lighting to break semantic alignment in VLMs such as CLIP, LLaVA, and BLIP, causing classification failures and hallucinations in real scenes.

  3. GeoFlowVLM: Geometry-Aware Joint Uncertainty for Frozen Vision-Language Embedding

    cs.LG 2026-05 unverdicted novelty 7.0

    GeoFlowVLM learns joint distributions of l2-normalized VLM embeddings on the product hypersphere via Riemannian flow matching to expose both aleatoric and epistemic uncertainty through derived entropy and typicality scores.

  4. Same Image, Different Meanings: Toward Retrieval of Context-Dependent Meanings

    cs.IR 2026-05 unverdicted novelty 7.0

    Image meanings grow more context-dependent with semantic abstraction, requiring narrative grounding for accurate retrieval at higher levels.

  5. jina-embeddings-v5-omni: Geometry-preserving Embeddings via Locked Aligned Towers

    cs.CL 2026-05 unverdicted novelty 7.0

    Jina-embeddings-v5-omni creates multimodal embeddings for text, image, audio, and video by freezing the text and media encoders and training only 0.35% of the weights via a VLM-style connector.

  6. Exploring Hierarchical Consistency and Unbiased Objectness for Open-Vocabulary Object Detection

    cs.CV 2026-04 unverdicted novelty 7.0

    Hierarchical confidence calibration and LoCLIP adaptation improve pseudo-label quality for open-vocabulary object detection, achieving new state-of-the-art results on COCO and LVIS benchmarks.

  7. BERAG: Bayesian Ensemble Retrieval-Augmented Generation for Knowledge-based Visual Question Answering

    cs.CL 2026-04 unverdicted novelty 7.0

    BERAG applies Bayesian ensemble weighting of individual documents via token-by-token posterior updates in retrieval-augmented generation, yielding gains on knowledge-based visual QA tasks.

  8. OVS-DINO: Open-Vocabulary Segmentation via Structure-Aligned SAM-DINO with Language Guidance

    cs.CV 2026-04 unverdicted novelty 7.0

    OVS-DINO structurally aligns DINO with SAM to revitalize attenuated boundary features, achieving SOTA gains of 2.1% average and 6.3% on Cityscapes in weakly-supervised open-vocabulary segmentation.

  9. Revealing Physical-World Semantic Vulnerabilities: Universal Adversarial Patches for Infrared Vision-Language Models

    cs.CV 2026-04 unverdicted novelty 7.0

    UCGP is a universal physical adversarial patch that compromises cross-modal semantic alignment in IR-VLMs through curved-grid parameterization and representation-space disruption.

  10. When Surfaces Lie: Exploiting Wrinkle-Induced Attention Shift to Attack Vision-Language Models

    cs.CV 2026-03 unverdicted novelty 7.0

    A wrinkle-field perturbation method creates photorealistic non-rigid image changes that degrade state-of-the-art VLMs on image captioning and VQA more effectively than prior baselines.

  11. SteelDefectX: A Multi-Form Vision-Language Dataset and Benchmark for Steel Surface Defect Analysis

    cs.CV 2026-03 unverdicted novelty 7.0

    SteelDefectX is a new multi-form vision-language dataset and benchmark for analyzing steel surface defects using 7,778 images across 25 categories.

  12. WikiCLIP: An Efficient Contrastive Baseline for Open-domain Visual Entity Recognition

    cs.CV 2026-03 unverdicted novelty 7.0

    WikiCLIP delivers an efficient contrastive baseline for open-domain visual entity recognition that improves accuracy by 16% on OVEN unseen entities and runs nearly 100 times faster than leading generative models.

  13. Mitigating Error Accumulation in Continuous Navigation via Memory-Augmented Kalman Filtering

    cs.RO 2026-01 unverdicted novelty 7.0

    NeuroKalman mitigates state drift in vision-language UAV navigation by using memory-augmented Kalman filtering where attention retrieves historical anchors to correct predictions without gradient updates.

  14. PowerCLIP: Powerset Alignment for Contrastive Pre-Training

    cs.CV 2025-11 conditional novelty 7.0

    PowerCLIP improves CLIP-style models by exhaustively aligning powersets of image regions to textual parse trees via efficient non-linear aggregators that approximate the full combinatorial loss.

  15. An Empirical Study of Validating Synthetic Data for Text-Based Person Retrieval

    cs.CV 2025-03 unverdicted novelty 7.0

    Empirical study of a fully synthetic data generation pipeline for text-based person retrieval that tests its use as a replacement or augmentation for real data across scenarios.

  16. Janus: Decoupling Visual Encoding for Unified Multimodal Understanding and Generation

    cs.CV 2024-10 unverdicted novelty 7.0

    Janus decouples visual encoding into task-specific pathways inside a single autoregressive transformer to unify multimodal understanding and generation while outperforming earlier unified models.

  17. VLM2Vec: Training Vision-Language Models for Massive Multimodal Embedding Tasks

    cs.CV 2024-10 conditional novelty 7.0

    VLM2Vec converts state-of-the-art vision-language models into universal multimodal embedders via contrastive training on the new MMEB benchmark, delivering 10-20% absolute gains over prior models on both in-distributi...

  18. Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs

    cs.CV 2024-06 unverdicted novelty 7.0

    Cambrian-1 is a vision-centric multimodal LLM family that evaluates over 20 vision encoders, introduces CV-Bench and the Spatial Vision Aggregator, and releases open models, code, and data achieving strong performance...

  19. MuirBench: A Comprehensive Benchmark for Robust Multi-image Understanding

    cs.CV 2024-06 conditional novelty 7.0

    MuirBench is a new benchmark showing that top multimodal LLMs struggle with robust multi-image understanding, with GPT-4o at 68% and open-source models below 33% accuracy.

  20. LVBench: An Extreme Long Video Understanding Benchmark

    cs.CV 2024-06 accept novelty 7.0

    LVBench is a new benchmark for extreme long video understanding that evaluates multimodal large language models on hour-scale videos using tasks designed to probe extended memory and comprehension.

  21. VideoChat: Chat-Centric Video Understanding

    cs.CV 2023-05 conditional novelty 7.0

    VideoChat integrates video models and LLMs via a learnable interface for chat-based spatiotemporal and causal video reasoning, trained on a new video-centric instruction dataset.

  22. UniRefiner: Teaching Pre-trained ViTs to Self-Dispose Dross via Contrastive Register

    cs.CV 2026-05 unverdicted novelty 6.0

    UniRefiner uses contrastive registers and a dual alignment objective to remove three categories of spurious tokens from pre-trained ViTs, yielding up to 9.4% mIoU gains on ADE20K and 22% zero-shot segmentation improvements.

  23. WOW-Seg: A Word-free Open World Segmentation Model

    cs.CV 2026-05 conditional novelty 6.0

    WOW-Seg proposes a word-free open-world segmentation model using Mask2Token and Cascade Attention Mask modules, reporting 89.7 semantic similarity and 82.4 semantic IoU on LVIS with one-eighth the parameters of prior ...

  24. Learning to See What You Need: Gaze Attention for Multimodal Large Language Models

    cs.CV 2026-05 unverdicted novelty 6.0

    Gaze Attention groups visual embeddings into selectable regions and dynamically restricts attention to task-relevant ones, matching dense baselines with up to 90% fewer visual KV entries via added context tokens.

  25. LLaVA-UHD v4: What Makes Efficient Visual Encoding in MLLMs?

    cs.CV 2026-05 unverdicted novelty 6.0

    LLaVA-UHD v4 reduces visual-encoding FLOPs by 55.8% for high-resolution images in MLLMs via slice-based encoding plus intra-ViT early compression while matching or exceeding baseline performance on document, OCR, and ...

  26. jina-embeddings-v5-omni: Geometry-preserving Embeddings via Locked Aligned Towers

    cs.CL 2026-05 unverdicted novelty 6.0

    GELATO extends frozen text embedding models with locked image and audio encoders, training minimal connectors to produce a single semantic embedding space for text, image, audio, and video while keeping original text ...

  27. Probing CLIP's Comprehension of 360-Degree Textual and Visual Semantics

    cs.CV 2026-04 conditional novelty 6.0

    CLIP models understand 360-degree textual semantics via explicit identifiers but show limited comprehension of visual semantics under horizontal circular shifts, which a LoRA fine-tuning approach improves with a noted...

  28. Rethinking Cross-Domain Evaluation for Face Forgery Detection with Semantic Fine-grained Alignment and Mixture-of-Experts

    cs.CV 2026-04 unverdicted novelty 6.0

    Cross-AUC exposes large robustness drops in existing face forgery detectors across datasets, while the SFAM model with semantic alignment and region-specific experts delivers better performance on public benchmarks.

  29. MiMIC: Mitigating Visual Modality Collapse in Universal Multimodal Retrieval While Avoiding Semantic Misalignment

    cs.CV 2026-04 unverdicted novelty 6.0

    MiMIC mitigates visual modality collapse and semantic misalignment in universal multimodal retrieval via fusion-in-decoder architecture and robust single-modality training.

  30. Exploring High-Order Self-Similarity for Video Understanding

    cs.CV 2026-04 unverdicted novelty 6.0

    The MOSS module learns and combines multi-order space-time self-similarity features to enhance temporal dynamics modeling in videos across action recognition, VQA, and robotic tasks.

  31. Cross-Attentive Multiview Fusion of Vision-Language Embeddings

    cs.CV 2026-04 unverdicted novelty 6.0

    CAMFusion fuses multiview 2D vision-language embeddings via cross-attention and multiview consistency self-supervision to produce better 3D semantic and instance representations, outperforming averaging and reaching S...

  32. Dual-Modality Anchor-Guided Filtering for Test-time Prompt Tuning

    cs.CV 2026-04 unverdicted novelty 6.0

    Dual-modality anchors from text descriptions and test-time image statistics filter views and ensemble predictions to improve test-time prompt tuning, achieving SOTA on 15 datasets.

  33. Chain-of-Models Pre-Training: Rethinking Training Acceleration of Vision Foundation Models

    cs.CV 2026-04 unverdicted novelty 6.0

    CoM-PT trains vision foundation models in ascending size order using inverse knowledge transfer, allowing larger models to achieve superior performance with significantly reduced overall computational cost compared to...

  34. TIPSv2: Advancing Vision-Language Pretraining with Enhanced Patch-Text Alignment

    cs.CV 2026-04 unverdicted novelty 6.0

    TIPSv2 improves dense patch-text alignment in vision-language pretraining through distillation and iBOT++ modifications, yielding models on par with or better than recent baselines on 9 tasks across 20 datasets.

  35. WikiSeeker: Rethinking the Role of Vision-Language Models in Knowledge-Based Visual Question Answering

    cs.CV 2026-04 unverdicted novelty 6.0

    WikiSeeker boosts KB-VQA performance by using VLMs to rewrite image-informed queries for better retrieval and to decide when to route to external LLM or rely on internal VLM knowledge.

  36. CoME-VL: Scaling Complementary Multi-Encoder Vision-Language Learning

    cs.CV 2026-04 unverdicted novelty 6.0

    CoME-VL fuses contrastive and self-supervised vision encoders via entropy-guided multi-layer aggregation and RoPE cross-attention to improve vision-language model performance on benchmarks.

  37. Contrastive Language-Colored Pointmap Pretraining for Unified 3D Scene Understanding

    cs.CV 2026-04 unverdicted novelty 6.0

    UniScene3D learns unified 3D scene representations from colored pointmaps using contrastive CLIP pretraining plus cross-view geometric and grounded view alignments, achieving state-of-the-art results on viewpoint grou...

  38. Vision Transformers Need More Than Registers

    cs.CV 2026-02 unverdicted novelty 6.0

    ViTs exhibit lazy aggregation by relying on irrelevant background patches for global semantics, and selectively integrating patch features into the CLS token reduces this effect and improves results across label-, tex...

  39. Modality Gap-Driven Subspace Alignment Training Paradigm For Multimodal Large Language Models

    cs.CV 2026-02 unverdicted novelty 6.0

    ReAlign corrects the modality gap in unpaired data to let MLLMs learn visual distributions from text alone before instruction tuning, reducing dependence on expensive paired corpora.

  40. CLAMP: Contrastive Learning for 3D Multi-View Action-Conditioned Robotic Manipulation Pretraining

    cs.RO 2026-01 unverdicted novelty 6.0

    CLAMP pretrains 3D multi-view encoders with contrastive learning on point clouds and actions, then initializes diffusion policies for more sample-efficient fine-tuning on robotic tasks.

  41. R3G: A Reasoning--Retrieval--Reranking Framework for Vision-Centric Answer Generation

    cs.CV 2026-01 unverdicted novelty 6.0

    R3G improves vision-centric visual question answering by generating reasoning plans to guide two-stage image retrieval and reranking, achieving state-of-the-art results on MRAG-Bench across six MLLM backbones.

  42. Calibrated Multimodal Representation Learning with Missing Modalities

    cs.CV 2025-11 unverdicted novelty 6.0

    CalMRL mitigates anchor shift in multimodal representation learning by calibrating incomplete alignments through representation-level imputation of missing modalities using priors and a bi-step optimization with close...

  43. VFM-VAE: Vision Foundation Models Can Be Good Tokenizers for Latent Diffusion Models

    cs.CV 2025-10 unverdicted novelty 6.0

    VFM-VAE uses a frozen VFM directly as LDM tokenizer via a custom decoder, reaching gFID 2.22 in 80 epochs and 1.62 after 640 epochs.

  44. FreeRet: MLLMs as Training-Free Retrievers

    cs.CV 2025-09 unverdicted novelty 6.0

    FreeRet enables pretrained MLLMs to act as training-free retrievers via semantically grounded embeddings and reasoning-based reranking, outperforming models trained on millions of pairs on MMEB benchmarks.

  45. Progressive Multimodal Search and Reasoning for Knowledge-Intensive Visual Question Answering

    cs.CV 2025-08 unverdicted novelty 6.0

    PMSR progressively constructs structured reasoning trajectories with dual-scope queries and compositional reasoning to improve knowledge acquisition and answer accuracy in knowledge-intensive VQA.

  46. Franca: Nested Matryoshka Clustering for Scalable Visual Representation Learning

    cs.CV 2025-07 conditional novelty 6.0

    Franca introduces nested Matryoshka clustering and positional disentanglement in a transparent SSL pipeline to deliver open-source vision models competitive with closed proprietary systems.

  47. Benchmarking Large Vision-Language Models on Fine-Grained Image Tasks: A Comprehensive Evaluation

    cs.CV 2025-04 unverdicted novelty 6.0

    Introduces FG-BMK benchmark and evaluates twelve LVLMs on fine-grained semantic recognition and feature tasks, identifying influences from training paradigms and perturbation sensitivity.

  48. Perception Encoder: The best visual embeddings are not at the output of the network

    cs.CV 2025-04 unverdicted novelty 6.0

    Intermediate layers of a contrastively trained vision-language encoder yield stronger general embeddings than the output layer, enabling state-of-the-art performance across image/video classification, multimodal QA, a...

  49. VLM-R1: A Stable and Generalizable R1-style Large Vision-Language Model

    cs.CV 2025-04 unverdicted novelty 6.0

    VLM-R1 applies R1-style RL using rule-based rewards on visual tasks with clear ground truth to achieve competitive performance and superior generalization over SFT in vision-language models.

  50. Grad-ECLIP: Gradient-based Visual and Textual Explanations for CLIP

    cs.CV 2025-02 conditional novelty 6.0

    Grad-ECLIP produces gradient-based visual and textual explanation heatmaps for CLIP by applying channel and spatial weights to token features instead of relying on sparse self-attention maps.

  51. MetaMorph: Multimodal Understanding and Generation via Instruction Tuning

    cs.CV 2024-12 unverdicted novelty 6.0

    VPiT enables pretrained LLMs to perform both visual understanding and generation by predicting discrete text tokens and continuous visual tokens, with understanding data proving more effective than generation-specific data.

  52. Uni-NaVid: A Video-based Vision-Language-Action Model for Unifying Embodied Navigation Tasks

    cs.RO 2024-12 unverdicted novelty 6.0

    Uni-NaVid unifies diverse embodied navigation tasks into one video-based vision-language-action model trained on 3.6 million samples from four sub-tasks, achieving state-of-the-art performance on benchmarks and real-w...

  53. E5-V: Universal Embeddings with Multimodal Large Language Models

    cs.CL 2024-07 unverdicted novelty 6.0

    E5-V produces strong universal multimodal embeddings from MLLMs trained solely on text pairs, often surpassing prior methods across retrieval and related tasks without multimodal fine-tuning.

  54. BLINK: Multimodal Large Language Models Can See but Not Perceive

    cs.CV 2024-04 accept novelty 6.0

    BLINK benchmark shows multimodal LLMs reach only 45-51 percent accuracy on core visual perception tasks where humans achieve 95 percent, indicating these abilities have not emerged.

  55. NaVid: Video-based VLM Plans the Next Step for Vision-and-Language Navigation

    cs.CV 2024-02 unverdicted novelty 6.0

    NaVid, a video-based VLM trained on 510k navigation and 763k web samples, achieves SOTA VLN performance using only monocular RGB video for next-step action planning in sim and real environments.

  56. MVBench: A Comprehensive Multi-modal Video Understanding Benchmark

    cs.CV 2023-11 accept novelty 6.0

    MVBench is a benchmark of 20 temporal video understanding tasks built by transforming static tasks into dynamic ones, with VideoChat2 outperforming prior MLLMs by over 15%.

  57. InternVid: A Large-scale Video-Text Dataset for Multimodal Understanding and Generation

    cs.CV 2023-07 unverdicted novelty 6.0

    InternVid supplies 7M videos and LLM captions to train ViCLIP, which reaches leading zero-shot action recognition and competitive retrieval performance.

  58. Sigmoid Loss for Language Image Pre-Training

    cs.CV 2023-03 conditional novelty 6.0

    SigLIP replaces softmax-based contrastive loss with a simple pairwise sigmoid loss for vision-language pre-training, decoupling batch size from normalization and reaching strong zero-shot performance with limited compute.

  59. What Matters for Grocery Product Retrieval with Open Source Vision Language Models

    cs.CV 2026-05 unverdicted novelty 5.0

    Systematic zero-shot benchmarking of open-source VLMs on multimodal grocery product retrieval shows data quality outperforms scale, introduces semantic power density as an efficiency metric, and identifies a persisten...

  60. AttenA+: Rectifying Action Inequality in Robotic Foundation Models

    cs.RO 2026-05 unverdicted novelty 5.0

    AttenA+ applies velocity-driven action attention to reweight training objectives toward kinematically critical low-velocity segments, yielding small benchmark gains on Libero and RoboTwin without added parameters.

Reference graph

Works this paper leans on

53 extracted references · 53 canonical work pages · cited by 72 Pith papers · 10 internal anchors

  1. [1]

    https://laion.ai/blog/giant-openclip/

    Reaching 80 zero-shot accuracy with openclip: Vit-g/14 trained on laion-2b. https://laion.ai/blog/giant-openclip/. 4

    work page

  2. [2]

    BEiT: BERT Pre-Training of Image Transformers

    Hangbo Bao, Li Dong, and Furu Wei. Beit: Bert pre-training of image transformers. arXiv preprint arXiv:2106.08254, 2021. 5, 6

    work page internal anchor Pith review arXiv 2021

  3. [3]

    Ob- jectnet: A large-scale bias-controlled dataset for pushing the limits of object recognition models

    Andrei Barbu, David Mayo, Julian Alverio, William Luo, Christo- pher Wang, Dan Gutfreund, Josh Tenenbaum, and Boris Katz. Ob- jectnet: A large-scale bias-controlled dataset for pushing the limits of object recognition models. In NeurIPS, 2019. 3

    work page 2019

  4. [4]

    Birdsnap: Large- 5 config EV A-01-CLIP-g / EV A-02-CLIP-g+ image enc

    Thomas Berg, Jiongxin Liu, Seung Woo Lee, Michelle L Alexan- der, David W Jacobs, and Peter N Belhumeur. Birdsnap: Large- 5 config EV A-01-CLIP-g / EV A-02-CLIP-g+ image enc. weight init. EV A-01-g text enc. weight init. Openai CLIP-L / Open CLIP-H image-text data LAION-400M [46] / Merged-2B image enc. peak learning rate 4e-4 image enc. layer-wise lr deca...

    work page 2014

  5. [5]

    Food- 101–mining discriminative components with random forests

    Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food- 101–mining discriminative components with random forests. In ECCV, 2014. 3

    work page 2014

  6. [6]

    Coyo-700m: Image- text pair dataset

    Minwoo Byeon, Beomhee Park, Haecheon Kim, Sungjun Lee, Woonhyuk Baek, and Saehoon Kim. Coyo-700m: Image- text pair dataset. https://github.com/kakaobrain/ coyo-dataset, 2022. 4

    work page 2022

  7. [7]

    A Short Note about Kinetics-600

    Joao Carreira, Eric Noland, Andras Banki-Horvath, Chloe Hillier, and Andrew Zisserman. A short note about kinetics-600. arXiv preprint arXiv:1808.01340, 2018. 4

    work page Pith review arXiv 2018

  8. [8]

    A short note on the kinetics-700 human action dataset.arXiv preprint arXiv:1907.06987, 2019

    Joao Carreira, Eric Noland, Chloe Hillier, and Andrew Zisserman. A short note on the kinetics-700 human action dataset. arXiv preprint arXiv:1907.06987, 2019. 4

    work page arXiv 1907

  9. [9]

    Quo vadis, action recogni- tion? a new model and the kinetics dataset

    Joao Carreira and Andrew Zisserman. Quo vadis, action recogni- tion? a new model and the kinetics dataset. In CVPR, 2017. 4

    work page 2017

  10. [10]

    Train- ing deep nets with sublinear memory cost, 2016

    Tianqi Chen, Bing Xu, Chiyuan Zhang, and Carlos Guestrin. Train- ing deep nets with sublinear memory cost, 2016. 4, 5

    work page 2016

  11. [11]

    Remote sensing im- age scene classification: Benchmark and state of the art

    Gong Cheng, Junwei Han, and Xiaoqiang Lu. Remote sensing im- age scene classification: Benchmark and state of the art. Proceed- ings of the IEEE, 2017. 3

    work page 2017

  12. [12]

    Cimpoi, S

    M. Cimpoi, S. Maji, I. Kokkinos, S. Mohamed, , and A. Vedaldi. Describing textures in the wild. In CVPR, 2014. 3

    work page 2014

  13. [13]

    ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators

    Kevin Clark, Minh-Thang Luong, Quoc V Le, and Christopher D Manning. ELECTRA: Pre-training text encoders as discriminators rather than generators. arXiv preprint arXiv:2003.10555, 2020. 5, 6

    work page internal anchor Pith review arXiv 2003

  14. [14]

    An analysis of single- layer networks in unsupervised feature learning

    Adam Coates, Andrew Ng, and Honglak Lee. An analysis of single- layer networks in unsupervised feature learning. In AISTAT, 2011. 3

    work page 2011

  15. [15]

    Fu, Stefano Ermon, Atri Rudra, and Christopher R´e

    Tri Dao, Daniel Y . Fu, Stefano Ermon, Atri Rudra, and Christopher R´e. Flashattention: Fast and memory-efficient exact attention with io-awareness, 2022. 1, 4

    work page 2022

  16. [16]

    Imagenet: A large-scale hierarchical image database

    Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei- Fei. Imagenet: A large-scale hierarchical image database. InCVPR,

    work page

  17. [17]

    An image is worth 16x16 words: Transformers for image recognition at scale, 2020

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa De- hghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale, 2020. 5

    work page 2020

  18. [18]

    The pascal visual object classes challenge: A retrospective

    Mark Everingham, SM Ali Eslami, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes challenge: A retrospective. IJCV, 2015. 3

    work page 2015

  19. [19]

    Eva-02: A visual representation for neon genesis,

    Yuxin Fang, Quan Sun, Xinggang Wang, Tiejun Huang, Xinlong Wang, and Yue Cao. Eva-02: A visual representation for neon genesis. arXiv preprint arXiv:2303.11331, 2023. 1, 2, 3, 4

    work page arXiv 2023

  20. [20]

    Eva: Exploring the limits of masked visual representation learning at scale

    Yuxin Fang, Wen Wang, Binhui Xie, Quan Sun, Ledell Wu, Xing- gang Wang, Tiejun Huang, Xinlong Wang, and Yue Cao. Eva: Ex- ploring the limits of masked visual representation learning at scale. arXiv preprint arXiv:2211.07636, 2022. 1, 2, 3

    work page arXiv 2022

  21. [21]

    Learning generative vi- sual models from few training examples: An incremental bayesian approach tested on 101 object categories

    Li Fei-Fei, Rob Fergus, and Pietro Perona. Learning generative vi- sual models from few training examples: An incremental bayesian approach tested on 101 object categories. In CVPRW, 2004. 3

    work page 2004

  22. [22]

    Challenges in repre- sentation learning: A report on three machine learning contests

    Ian J Goodfellow, Dumitru Erhan, Pierre Luc Carrier, Aaron Courville, Mehdi Mirza, Ben Hamner, Will Cukierski, Yichuan Tang, David Thaler, Dong-Hyun Lee, et al. Challenges in repre- sentation learning: A report on three machine learning contests. In ICONIP, 2013. 3

    work page 2013

  23. [23]

    Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification.IEEE J

    Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification.IEEE J. Sel. Top. Appl. Earth Obs. Remote Sens., 2019. 3

    work page 2019

  24. [24]

    The many faces of robustness: A critical analysis of out-of-distribution generalization

    Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Para- juli, Mike Guo, et al. The many faces of robustness: A critical analysis of out-of-distribution generalization. In CVPR, 2021. 3

    work page 2021

  25. [25]

    Natural adversarial examples

    Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adversarial examples. In CVPR, 2021. 3

    work page 2021

  26. [26]

    Deep networks with stochastic depth

    Gao Huang, Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Q Wein- berger. Deep networks with stochastic depth. In ECCV, 2016. 5, 6

    work page 2016

  27. [27]

    Openclip

    Gabriel Ilharco, Mitchell Wortsman, Ross Wightman, Cade Gor- don, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongseok Namkoong, John Miller, Hannaneh Hajishirzi, Ali Farhadi, and Ludwig Schmidt. Openclip. https://github. com/mlfoundations/open_clip, 2021. 3 6

    work page 2021

  28. [28]

    Adam: A Method for Stochastic Optimization

    Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014. 6

    work page internal anchor Pith review Pith/arXiv arXiv 2014

  29. [29]

    3d object representations for fine-grained categorization

    Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine-grained categorization. In ICCVW,

    work page

  30. [30]

    Learning multiple layers of features from tiny images

    Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009. 3

    work page 2009

  31. [31]

    Gradient-based learning applied to document recognition.Proceed- ings of the IEEE, 1998

    Yann LeCun, L ´eon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition.Proceed- ings of the IEEE, 1998. 3

    work page 1998

  32. [32]

    BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip- 2: Bootstrapping language-image pre-training with frozen im- age encoders and large language models. arXiv preprint arXiv:2301.12597, 2023. 1

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  33. [33]

    Scaling language-image pre-training via masking,

    Yanghao Li, Haoqi Fan, Ronghang Hu, Christoph Feichtenhofer, and Kaiming He. Scaling language-image pre-training via masking,

    work page

  34. [34]

    Mi- crosoft coco: Common objects in context

    Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll´ar, and C Lawrence Zitnick. Mi- crosoft coco: Common objects in context. In European conference on computer vision, pages 740–755. Springer, 2014. 4

    work page 2014

  35. [35]

    Decoupled weight decay regu- larization

    Ilya Loshchilov and Frank Hutter. Decoupled weight decay regu- larization. In ICLR, 2019. 5, 6

    work page 2019

  36. [36]

    Fine-Grained Visual Classification of Aircraft

    Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine-grained visual classification of aircraft.arXiv preprint arXiv:1306.5151, 2013. 3

    work page internal anchor Pith review Pith/arXiv arXiv 2013

  37. [37]

    Automated flower classification over a large number of classes

    Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In ICVGIP, 2008. 3

    work page 2008

  38. [38]

    Parkhi, Andrea Vedaldi, Andrew Zisserman, and C

    Omkar M. Parkhi, Andrea Vedaldi, Andrew Zisserman, and C. V . Jawahar. Cats and dogs. In CVPR, 2012. 3

    work page 2012

  39. [39]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML, 2021. 1, 3

    work page 2021

  40. [40]

    Zero: Memory optimizations toward training trillion parameter models

    Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. Zero: Memory optimizations toward training trillion parameter models. In SC20, 2020. 2, 4, 5

    work page 2020

  41. [41]

    Hierarchical Text-Conditional Image Generation with CLIP Latents

    Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125, 2022. 1

    work page internal anchor Pith review Pith/arXiv arXiv 2022

  42. [42]

    Zero-Shot Text-to-Image Generation

    Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea V oss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. arXiv preprint arXiv:2102.12092, 2021. 5, 6

    work page internal anchor Pith review arXiv 2021

  43. [43]

    Deepspeed: System optimizations enable training deep learn- ing models with over 100 billion parameters

    Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. Deepspeed: System optimizations enable training deep learn- ing models with over 100 billion parameters. In KDD, 2020. 2, 4, 5, 6

    work page 2020

  44. [44]

    Do imagenet classifiers generalize to imagenet?, 2019

    Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to imagenet?, 2019. 3

    work page 2019

  45. [45]

    LAION-5B: An open large-scale dataset for training next generation image-text models

    Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion-5b: An open large-scale dataset for training next generation image-text models. arXiv preprint arXiv:2210.08402, 2022. 1, 4, 6

    work page internal anchor Pith review arXiv 2022

  46. [46]

    LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairs

    Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Je- nia Jitsev, and Aran Komatsuzaki. Laion-400m: Open dataset of clip-filtered 400 million image-text pairs. arXiv preprint arXiv:2111.02114, 2021. 5, 6

    work page internal anchor Pith review Pith/arXiv arXiv 2021

  47. [47]

    UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild

    Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. Ucf101: A dataset of 101 human actions classes from videos in the wild. arXiv preprint arXiv:1212.0402, 2012. 4

    work page internal anchor Pith review Pith/arXiv arXiv 2012

  48. [48]

    Johannes Stallkamp, Marc Schlipsing, Jan Salmen, and Christian Igel. Man vs. computer: Benchmarking machine learning algo- rithms for traffic sign recognition. Neural networks, 2012. 3

    work page 2012

  49. [49]

    Rotation equivariant cnns for digital pathology

    Bastiaan S Veeling, Jasper Linmans, Jim Winkens, Taco Cohen, and Max Welling. Rotation equivariant cnns for digital pathology. In MICCAI, 2018. 3

    work page 2018

  50. [50]

    Learning robust global representations by penalizing local predic- tive power

    Haohan Wang, Songwei Ge, Zachary Lipton, and Eric P Xing. Learning robust global representations by penalizing local predic- tive power. NeurIPS, 2019. 3

    work page 2019

  51. [51]

    Sun database: Large-scale scene recognition from abbey to zoo

    Jianxiong Xiao, James Hays, Krista A Ehinger, Aude Oliva, and Antonio Torralba. Sun database: Large-scale scene recognition from abbey to zoo. In CVPR, 2010. 3

    work page 2010

  52. [52]

    Large batch optimization for deep learning: Training bert in 76 minutes, 2019

    Yang You, Jing Li, Sashank Reddi, Jonathan Hseu, Sanjiv Ku- mar, Srinadh Bhojanapalli, Xiaodan Song, James Demmel, Kurt Keutzer, and Cho-Jui Hsieh. Large batch optimization for deep learning: Training bert in 76 minutes, 2019. 1, 3, 5, 6

    work page 2019

  53. [53]

    From image descriptions to visual denotations: New similarity met- rics for semantic inference over event descriptions

    Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier. From image descriptions to visual denotations: New similarity met- rics for semantic inference over event descriptions. TACL, 2014. 4 7

    work page 2014