Pith. sign in

REVIEW 38 cited by

LLaVAR: Enhanced Visual Instruction Tuning for Text-Rich Image Understanding

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2306.17107 v2 pith:JOXEDJ3G submitted 2023-06-29 cs.CV cs.CL

Yanzhe Zhang , Ruiyi Zhang , Jiuxiang Gu , Yufan Zhou , Nedim Lipka , Diyi Yang , Tong Sun This is my paper · ORCID
classification cs.CVcs.CL
keywords imagestext-richllavarvisualdatainstructioninstruction-followingmodel
verification ladder T0 review T1 audit T2 compute T3 formal

Signed reviews

No signed human review yet.

0 comments
read the original abstract

Instruction tuning unlocks the superior capability of Large Language Models (LLM) to interact with humans. Furthermore, recent instruction-following datasets include images as visual inputs, collecting responses for image-based instructions. However, visual instruction-tuned models cannot comprehend textual details within images well. This work enhances the current visual instruction tuning pipeline with text-rich images (e.g., movie posters, book covers, etc.). Specifically, we first use publicly available OCR tools to collect results on 422K text-rich images from the LAION dataset. Moreover, we prompt text-only GPT-4 with recognized texts and image captions to generate 16K conversations, each containing question-answer pairs for text-rich images. By combining our collected data with previous multi-modal instruction-following data, our model, LLaVAR, substantially improves the LLaVA model's capability on text-based VQA datasets (up to 20% accuracy improvement) while achieving an accuracy of 91.42% on ScienceQA. The GPT-4-based instruction-following evaluation also demonstrates the improvement of our model on both natural images and text-rich images. Through qualitative analysis, LLaVAR shows promising interaction (e.g., reasoning, writing, and elaboration) skills with humans based on the latest real-world online content that combines text and images. We make our code/data/models publicly available at https://llavar.github.io/.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 38 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. PRISM: Priority-aware Rubric Internalization via Structured Multimodal Data Synthesis

    cs.LG 2026-08 conditional novelty 7.0 of 10

    PRISM trains MLLMs to act as rubric executors by synthesizing typed, prioritized rules and verification traces, lifting Qwen3-VL-4B from 9.5% to 30.1% Strict accuracy on the authors' PRISM-Eval benchmark.

  2. Synthetic Visual Genome

    cs.CV 2025-06 conditional novelty 7.0 of 10

    A GPT-4V/GPT-4o pipeline for completing and refining scene graph annotations yields a dense synthetic dataset that, after instruction tuning, gives a 3B model strong relationship understanding and grounding results.

  3. Mirage in the Eyes: Hallucination Attack on Multi-modal Large Language Models with Only Attention Sink

    cs.LG 2025-01 conditional novelty 7.0 of 10

    Adversarial images optimized to induce attention sink behavior increase hallucination rates in multiple MLLMs, including commercial APIs, without visibly degrading response quality.

  4. MonkeyOCRv2: A Visual-Text Foundation Model for Document AI

    cs.CV 2026-07 conditional novelty 6.5 of 10

    A document-oriented ViT family pretrained with text generation plus pixel reconstruction on 113M images transfers across recognition, detection, parsing, and understanding, setting open-source SOTA on MDPBench with a ...

  5. Twins: Learn to Predict Unified Representations with Focal Loss

    cs.CV 2026-07 conditional novelty 6.0 of 10

    Channel-wise concatenation of SigLIP2 and Flux VAE features into one token, trained with a focal-style flow-matching loss, yields a unified representation with 1.59 gFID on ImageNet 256 and VAE-level reconstruction.

  6. CausalStep: A Benchmark for Explicit Stepwise Causal Reasoning in Videos

    cs.CV 2025-07 conditional novelty 6.0 of 10

    CausalStep introduces a stepwise video QA protocol and reports that top multimodal models (chain success rate 51%) remain far below human performance (79%) on explicit causal chains.

  7. VDInstruct: Zero-Shot Key Information Extraction via Content-Aware Vision Tokenization

    cs.CV 2025-07 conditional novelty 6.0 of 10

    VDInstruct achieves strong zero-shot key-information extraction by combining a region detector with content-aware vision tokenization, using about 500 image tokens per page.

  8. Advancing Multimodal LLMs by Large-Scale 3D Visual Instruction Dataset Generation

    cs.GR 2025-07 conditional novelty 6.0 of 10

    Fine-tuning open vision-language models on 240K synthetic question-answer pairs with exact camera-object labels improves camera-object recognition by 33.4% on average over GPT-4o and Claude-3-Sonnet on the paper's benchmark.

  9. Single-to-mix Modality Alignment with Multimodal Large Language Model for Document Image Machine Translation

    cs.CL 2025-07 conditional novelty 6.0 of 10

    M4Doc distills the multimodal representations of a frozen MLLM into an image-only encoder, improving document image translation quality and generalization without requiring the MLLM at inference.

  10. Multimodal Mathematical Reasoning with Diverse Solving Perspective

    cs.CL 2025-07 conditional novelty 6.0 of 10

    Training a multimodal language model on multiple diverse solution paths per problem, plus rewards for distinguishing correct from incorrect solutions, improves math benchmark accuracy and output diversity.

  11. GenRecal: Generation after Recalibration from Large to Small Vision-Language Models

    cs.CL 2025-06 conditional novelty 6.0 of 10

    A learnable Recalibrator bridges different tokenizers so that small VLMs can distill knowledge from any large VLM, improving their benchmark scores.

  12. CoMemo: LVLMs Need Image Context with Image Memory

    cs.CV 2025-06 conditional novelty 6.0 of 10

    CoMemo adds a cross-attention image-memory path and thumbnail-anchored position encoding to reduce visual neglect in long-context and multi-image LVLM tasks.

  13. Omni-R1: Reinforcement Learning for Omnimodal Reasoning via Two-System Collaboration

    cs.CV 2025-05 conditional novelty 6.0 of 10

    Omni-R1 uses GRPO reinforcement learning to train a global reasoning model that selects keyframes and rewrites queries for a detail model, improving video and audio-visual segmentation and out-of-domain QA.

  14. FUDOKI: Discrete Flow-based Unified Understanding and Generation via Kinetic-Optimal Velocities

    cs.CV 2025-05 conditional novelty 6.0 of 10

    A 1.5B unified multimodal model trained with discrete flow matching and metric-induced probability paths matches autoregressive baselines of similar size on generation and understanding benchmarks.

  15. WildDoc: How Far Are We from Achieving Comprehensive and Robust Document Understanding in the Wild?

    cs.CV 2025-05 conditional novelty 6.0 of 10

    The paper introduces WildDoc, a photo-based document understanding benchmark, and shows that current vision-language models drop sharply in both accuracy and consistency on real-world captures.

  16. DocVXQA: Context-Aware Visual Explanations for Document Question Answering

    cs.CV 2025-05 conditional novelty 6.0 of 10

    DocVXQA learns sparsity-regularized masks guided by ColPali so a Pix2Struct-based DocVQA model answers from masked document images and thereby produces visual explanations.

  17. Eagle 2: Building Post-Training Data Strategies from Scratch for Frontier Vision-Language Models

    cs.CV 2025-01 conditional novelty 6.0 of 10

    Eagle2-9B matches or outperforms much larger vision-language models on many benchmarks through a carefully constructed post-training data strategy.

  18. A High-Quality Text-Rich Image Instruction Tuning Dataset via Hybrid Instruction Generation

    cs.CV 2024-12 conditional novelty 6.0 of 10

    A hybrid human/GPT-4o pipeline plus two new filtering scores produces a 424k-pair text-rich image instruction dataset that improves fine-tuned multimodal models on OCR and document understanding benchmarks.

  19. HoVLE: Unleashing the Power of Monolithic Vision-Language Models with Holistic Vision-Language Embedding

    cs.CV 2024-12 conditional novelty 6.0 of 10

    HoVLE is a monolithic VLM whose holistic embedding module maps images and text into one shared space, letting a frozen LLM reach near-compositional performance.

  20. MedMax: Mixed-Modal Instruction Tuning for Training Biomedical Assistants

    cs.AI 2024-12 conditional novelty 6.0 of 10

    A new 1.47M-instance biomedical instruction-tuning dataset improves a mixed-modal 7B model's medical VQA accuracy by 18 to 26 percentage points over GPT-4o and Chameleon.

  21. PVC: Progressive Visual Token Compression for Unified Image and Video Processing in Large Vision-Language Models

    cs.CV 2024-12 conditional novelty 6.0 of 10

    PVC unifies image and video token compression in VLMs by repeating images as static videos and using causal temporal attention with adaptive compression, achieving strong benchmark results at 64 tokens per frame.

  22. Chimera: Improving Generalist Model with Domain-Specific Experts

    cs.CV 2024-12 conditional novelty 6.0 of 10

    Chimera fuses frozen domain-expert encoders into a generalist multimodal LLM via routing and a 30% masking of general tokens, lifting InternVL2-8B from 61.6 to 64.9 on MathVista and from 31.3 to 32.4 on MathVerse.

  23. Enhancing Perception Capabilities of Multimodal LLMs with Training-Free Fusion

    cs.CV 2024-12 conditional novelty 6.0 of 10

    Concatenating visual tokens from multiple MLLMs in the same family and linearly merging their language model deltas yields a training-free fusion that improves multimodal benchmark scores over each source model.

  24. MIRe: Enhancing Multimodal Queries Representation via Fusion-Free Modality Interaction for Multimodal Retrieval

    cs.CV 2024-11 conditional novelty 6.0 of 10

    A multimodal retriever aligns images to text by allowing text tokens to attend to visual patches while excluding text embeddings from the trained visual representation, then combines both at scoring time.

  25. ET-Prune: Evidence-Aware Dynamic Budgeting for Visual Token Pruning in Text-Rich MLLMs

    cs.CV 2026-08 conditional novelty 5.0 of 10

    A question-aware, per-image budget decides which visual tokens to keep, letting OCR-focused multimodal models hold accuracy at about half the tokens.

  26. Beyond Emotion Recognition: A Multi-Turn Multimodal Emotion Understanding and Reasoning Benchmark

    cs.CV 2025-08 conditional novelty 5.0 of 10

    MTMEUR is a new multimodal emotion reasoning benchmark where the best single model scores 71.19% and a four-agent reasoning framework tops 72.93%.

  27. Argus: Vision-Centric Reasoning with Grounded Chain-of-Thought

    cs.CV 2025-05 conditional novelty 5.0 of 10

    Argus adds explicit language-guided visual attention to multimodal LLMs by grounding questions to bounding boxes and re-engaging those regions, improving vision-centric reasoning and grounding accuracy.

  28. Instructify: Demystifying Metadata to Visual Instruction Tuning Data Conversion

    cs.CV 2025-05 conditional novelty 5.0 of 10

    Instructify converts image metadata into visual instruction-tuning conversations with open LLMs, matching or exceeding GPT-4-generated data quality on LMM benchmarks.

  29. Eve: Efficient Multimodal Vision Language Models with Elastic Visual Experts

    cs.CV 2025-01 conditional novelty 5.0 of 10

    A 1.8B-parameter vision-language model, Eve, uses elastic visual experts and type-aware token routing to reach a 68.87% average on six VLM benchmarks while preserving language performance.

  30. Task Preference Optimization: Improving Multimodal Large Language Models with Vision Task Alignment

    cs.CV 2024-12 conditional novelty 5.0 of 10

    Adding task-specific heads for tracking, grounding, and segmentation to multimodal LLMs via a three-stage training recipe improves both fine-grained visual tasks and general video understanding benchmarks.

  31. FILA: Fine-Grained Vision Language Models

    cs.CV 2024-12 conditional novelty 5.0 of 10

    A new hybrid visual encoder with multi-layer ConvNeXt-ViT fusion improves high-resolution image understanding in vision-language models.

  32. DLaVA: Document Language and Vision Assistant for Answer Localization with Enhanced Interpretability and Trustworthiness

    cs.CV 2024-11 reject novelty 5.0 of 10

    By feeding a montage of text crops with numbered boxes to a vision-language model, DLaVA localizes answers in documents and reports stronger accuracy and localization than the baselines it compares against.

  33. Efficiently Integrate Large Language Models with Visual Perception: A Survey from the Training Paradigm Perspective

    cs.CV 2025-02 conditional novelty 4.0 of 10

    A taxonomy and benchmark comparison of 34 vision large language models, showing two-stage instruction tuning gives top scores while direct adapter tuning is most parameter-efficient, though the comparison is confounde...

  34. `Do as I say not as I do': A Semi-Automated Approach for Jailbreak Prompt Attack against Multimodal LLMs

    cs.CR 2025-02 reject novelty 4.0 of 10

    A voice jailbreak that buries a forbidden question between benign prompts reportedly succeeds against Gemini 67 to 93 percent of the time, but the metric comes from the target model judging itself and is not reliable.

  35. EACO: Enhancing Alignment in Multimodal LLMs via Critical Observation

    cs.CV 2024-12 conditional novelty 4.0 of 10

    A critic trained on filtered GPT-4V scoring data selects the best and worst self-generated responses, and DPO with extra SFT improves hallucination and reasoning benchmarks in multimodal LLMs.

  36. BlueLM-V-3B: Algorithm and System Co-Design for Multimodal Large Language Models on Mobile Devices

    cs.CV 2024-11 conditional novelty 4.0 of 10

    BlueLM-V-3B is a 3B multimodal language model that runs on a phone NPU at 24.4 token/s and reports strong visual benchmark scores, enabled by a relaxed image-resolution rule and hardware-aware inference.

  37. Visual Large Language Models for Generalized and Specialized Applications

    cs.CV 2025-01 conditional novelty 3.0 of 10

    This paper reviews and taxonomizes VLLM applications into vision-to-text, vision-to-action, and text-to-vision, adding ethics and future-work discussion.

  38. Survey on Question Answering over Visually Rich Documents: Methods, Challenges, and Trends

    cs.CL 2025-01 conditional novelty 3.0 of 10

    A structured overview of question answering over visually rich documents, comparing encoding, vision-only, and multi-page methods, and highlighting comparability issues in existing benchmarks.

Pith tools