Pith. sign in

REVIEW 24 cited by

VILA: On Pre-training for Visual Language Models

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2312.07533 v4 pith:EJZD36WT submitted 2023-12-12 cs.CV

Ji Lin , Hongxu Yin , Wei Ping , Yao Lu , Pavlo Molchanov , Andrew Tao , Huizi Mao , Jan Kautz
show 2 more authors
Mohammad Shoeybi Song Han
This is my paper · ORCID
classification cs.CV
keywords pre-trainingvisuallanguagemodelsdatainstructionviladuring
verification ladder T0 review T1 audit T2 compute T3 formal

Signed reviews

No signed human review yet.

0 comments
read the original abstract

Visual language models (VLMs) rapidly progressed with the recent success of large language models. There have been growing efforts on visual instruction tuning to extend the LLM with visual inputs, but lacks an in-depth study of the visual language pre-training process, where the model learns to perform joint modeling on both modalities. In this work, we examine the design options for VLM pre-training by augmenting LLM towards VLM through step-by-step controllable comparisons. We introduce three main findings: (1) freezing LLMs during pre-training can achieve decent zero-shot performance, but lack in-context learning capability, which requires unfreezing the LLM; (2) interleaved pre-training data is beneficial whereas image-text pairs alone are not optimal; (3) re-blending text-only instruction data to image-text data during instruction fine-tuning not only remedies the degradation of text-only tasks, but also boosts VLM task accuracy. With an enhanced pre-training recipe we build VILA, a Visual Language model family that consistently outperforms the state-of-the-art models, e.g., LLaVA-1.5, across main benchmarks without bells and whistles. Multi-modal pre-training also helps unveil appealing properties of VILA, including multi-image reasoning, enhanced in-context learning, and better world knowledge.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 24 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Q-GeoMem: Question-Guided Geometric Memory for Video Spatial Reasoning

    cs.CV 2026-05 unverdicted novelty 6.0 of 10

    Question-guided dual geometric memories with relevance-novelty utility reportedly reach state-of-the-art video spatial reasoning on two in-domain and five out-of-distribution benchmarks.

  2. SpaceTools: Tool-Augmented Spatial Reasoning via Double Interactive RL

    cs.CV 2025-12 conditional novelty 6.0 of 10

    A two-phase interactive RL framework (DIRL) lets a 3B VLM learn to coordinate multiple vision and robot tools, reaching top benchmark scores and 86% real-robot pick-and-place success.

  3. Video Reasoning without Training

    cs.CV 2025-10 conditional novelty 6.0 of 10

    An entropy-guided, inference-time value-cache controller improves video reasoning accuracy and cuts output tokens versus RL-trained baselines.

  4. Estimating the Empowerment of Language Model Agents

    cs.AI 2025-09 conditional novelty 6.0 of 10

    EELMA estimates the mutual information between an LM agent's actions and future text states, and this 'empowerment' is shown to correlate with task performance across toy games and WebArena.

  5. MESH -- Understanding Videos Like Human: Measuring Hallucinations in Large Video Models

    cs.CV 2025-09 conditional novelty 6.0 of 10

    MESH, a three-layer video hallucination benchmark, shows LVMs ace basic objects and coarse traits but slip badly on fine character details and multi-subject actions in longer clips.

  6. OwlCap: Harmonizing Motion-Detail for Video Captioning via HMD-270K and Caption Set Equivalence Reward

    cs.CV 2025-08 conditional novelty 6.0 of 10

    OwlCap balances motion and detail in video captions using the HMD-270K dataset and a Caption Set Equivalence Reward, improving VDC accuracy and DREAM-1K recall over baselines.

  7. AuroraLong: Bringing RNNs Back to Efficient Open-Ended Video Understanding

    cs.CV 2025-07 conditional novelty 6.0 of 10

    A 2B-parameter video-language model using an RWKV linear-RNN backbone and sorted token merging achieves competitive long-video QA accuracy with far lower memory cost than transformer-based models.

  8. GenRecal: Generation after Recalibration from Large to Small Vision-Language Models

    cs.CL 2025-06 conditional novelty 6.0 of 10

    A learnable Recalibrator bridges different tokenizers so that small VLMs can distill knowledge from any large VLM, improving their benchmark scores.

  9. Affordance Benchmark for MLLMs

    cs.CL 2025-06 conditional novelty 6.0 of 10

    A new 2,000-question benchmark finds multimodal AI models recognize object affordances far worse than humans, with top model Gemini-2.0-Pro at 18.05% versus 85.34% human best.

  10. Multi-modal Synthetic Data Training and Model Collapse: Insights from VLMs and Diffusion Models

    cs.LG 2025-05 conditional novelty 6.0 of 10

    Recursive training on synthetic data in multi-modal VLM and diffusion systems shows distinct collapse: caption variance grows while image variance shrinks, and frozen-model relabeling mitigates it.

  11. Weaving Context Across Images: Improving Vision-Language Models through Focus-Centric Visual Chains

    cs.CV 2025-04 conditional novelty 6.0 of 10

    A focus-centric reasoning format plus a 150K synthetic dataset improves VLM accuracy across seven multi-image benchmarks.

  12. VideoVista-CulturalLingo: 360$^\circ$ Horizons-Bridging Cultures, Languages, and Domains in Video Comprehension

    cs.CV 2025-04 conditional novelty 6.0 of 10

    VideoVista-CulturalLingo evaluates 24 video AI models on a bilingual, multicultural video QA benchmark and finds that models handle Western content better than Chinese cultural content, struggle with temporal localiza...

  13. Video-MMLU: A Massive Multi-Discipline Lecture Understanding Benchmark

    cs.CV 2025-04 conditional novelty 6.0 of 10

    Video-MMLU: a 1,065-video lecture benchmark where most AI video models score 10-50%, but text-only models answer 40% of quiz questions without video.

  14. AdaFV: Rethinking of Visual-Language alignment for VLM acceleration

    cs.CV 2025-01 conditional novelty 6.0 of 10

    AdaFV prunes visual tokens before the LLM by adaptively mixing visual saliency and text-to-image similarity, achieving state-of-the-art training-free VLM acceleration at high reduction rates.

  15. J-EDI QA: Benchmark for deep-sea organism-specific multimodal LLM

    cs.CV 2024-12 conditional novelty 6.0 of 10

    J-EDI QA is a new 100-image Japanese multiple-choice benchmark for deep-sea organism identification; OpenAI o1 scored 50%, GPT-4o 39%, and non-expert humans about 40%.

  16. Decoupled Visual Processing: Efficient Multimodal Adaptation via Modality-Specific Transformer Substitution

    cs.CV 2026-07 reject novelty 5.0 of 10

    The paper's proposed decoupled visual pathway, as described, would receive no gradient from the loss, so the central claim is not supported.

  17. MagicVL-2B: Empowering Vision-Language Models on Mobile Devices with Lightweight Visual Encoders via Curriculum Learning

    cs.CV 2025-08 unverdicted novelty 5.0 of 10

    MagicVL-2B is a 2B vision-language model for mobile phones that claims state-of-the-art-matching accuracy at 41.1% lower on-device power, via a lightweight encoder, dynamic resolution, and curriculum learning.

  18. Accelerating Multimodal Large Language Models via Dynamic Visual-Token Exit and the Empirical Findings

    cs.CV 2024-11 conditional novelty 5.0 of 10

    DyVTE trains a lightweight hyper-network to detect when text tokens have absorbed enough image information, then removes all visual tokens at that layer, cutting MLLM FLOPs by 20-56% with small accuracy changes.

  19. MLAN: Language-Based Instruction Tuning Preserves and Transfers Knowledge in Multimodal Language Models

    cs.CL 2024-11 conditional novelty 5.0 of 10

    Training a multimodal LLM on mostly text-only instructions with a small vision-language tail matches or beats vision-heavy instruction tuning on held-out text and vision tasks at about half the token cost.

  20. BcQLM: Efficient Vision-Language Understanding with Distilled Q-Gated Cross-Modal Fusion

    cs.CV 2025-09 conditional novelty 4.0 of 10

    A roughly 1.2B-parameter VQA model with a distilled 31M CLIP encoder and Q-gated cross-attention reports accuracies comparable to 7B-13B baselines on GQA, VQAv2, and VizWiz.

  21. Progressive Semantic Residual Quantization for Multimodal-Joint Interest Modeling in Music Recommendation

    cs.IR 2025-08 conditional novelty 4.0 of 10

    A progressive residual quantization that feeds the previous nearest centroid back into the codebook input, paired with a multi-codebook cross-attention network, yields small AUC gains in music recommendation.

  22. KptLLM++: Towards Generic Keypoint Comprehension with Large Language Model

    cs.CV 2025-07 conditional novelty 4.0 of 10

    KptLLM++ unifies keypoint semantic understanding, visual-prompt detection, and text-prompt detection in a single multimodal LLM, reporting SOTA accuracy on COCO, AP-10K, Human-Art, and other benchmarks.

  23. A Survey on Autonomy-Induced Security Risks in Large Model-Based Agents

    cs.AI 2025-06 conditional novelty 4.0 of 10

    The paper surveys security risks of LLM agents, organizes them into a five-level autonomy taxonomy, and proposes an untested CMDP-based architecture called R2A2.

  24. Vision-Language Models for Edge Networks: A Comprehensive Survey

    cs.CV 2025-02 reject novelty 2.0 of 10

    A survey of lightweight vision-language models for edge deployment, marred by citation errors, self-citation, and a lack of selection methodology.

Pith tools