Pith. sign in

REVIEW 35 cited by

SEED-Bench-2-Plus: Benchmarking Multimodal Large Language Models with Text-Rich Visual Comprehension

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2404.16790 v1 pith:YZI5SG7S submitted 2024-04-25 cs.CV

Bohao Li , Yuying Ge , Yi Chen , Yixiao Ge , Ruimao Zhang , Ying Shan This is my paper
classification cs.CV
keywords text-richmllmsvisualcomprehensionscenariosbenchmarkbenchmarkscategories
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Comprehending text-rich visual content is paramount for the practical application of Multimodal Large Language Models (MLLMs), since text-rich scenarios are ubiquitous in the real world, which are characterized by the presence of extensive texts embedded within images. Recently, the advent of MLLMs with impressive versatility has raised the bar for what we can expect from MLLMs. However, their proficiency in text-rich scenarios has yet to be comprehensively and objectively assessed, since current MLLM benchmarks primarily focus on evaluating general visual comprehension. In this work, we introduce SEED-Bench-2-Plus, a benchmark specifically designed for evaluating \textbf{text-rich visual comprehension} of MLLMs. Our benchmark comprises 2.3K multiple-choice questions with precise human annotations, spanning three broad categories: Charts, Maps, and Webs, each of which covers a wide spectrum of text-rich scenarios in the real world. These categories, due to their inherent complexity and diversity, effectively simulate real-world text-rich environments. We further conduct a thorough evaluation involving 34 prominent MLLMs (including GPT-4V, Gemini-Pro-Vision and Claude-3-Opus) and emphasize the current limitations of MLLMs in text-rich visual comprehension. We hope that our work can serve as a valuable addition to existing MLLM benchmarks, providing insightful observations and inspiring further research in the area of text-rich visual comprehension with MLLMs. The dataset and evaluation code can be accessed at https://github.com/AILab-CVC/SEED-Bench.

Discussion (0). Sign in to comment.

Forward citations

Cited by 35 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. DataComp-VLM: Improved Open Datasets for Vision-Language Models

    cs.CV 2026-06 conditional novelty 8.0 of 10

    DataComp-VLM benchmark shows instruction-heavy data mixing outperforms filtering for VLM training, with DCVLM-Baseline achieving 63.6% on 33 tasks for 8B models (+5.4pp over FineVision).

  2. MME-RealWorld: Could Your Multimodal LLM Challenge High-Resolution Real-World Scenarios that are Difficult for Humans?

    cs.CV 2024-08 conditional novelty 8.0 of 10

    MME-RealWorld is the largest manually annotated high-resolution benchmark for MLLMs, where even the best models achieve less than 60% accuracy on challenging real-world tasks.

  3. Contextualized Evaluation of Vision Language Models through Dynamic, Multi-turn Interactions

    cs.AI 2026-07 conditional novelty 7.0 of 10

    Context-guided multi-turn interviews reveal more VLM hallucinations than static benchmarks, and those hallucinations increase with conversational history and false-premise questions.

  4. CapRL++: Unified Reinforcement Learning with Verifiable Rewards for Dense Image and Video Captioning

    cs.CV 2026-06 unverdicted novelty 7.0 of 10

    CapRL++ applies reinforcement learning with verifiable rewards to dense image and video captioning by scoring captions via the accuracy of a vision-free LLM answering MCQs from the caption alone.

  5. HyLaR: Hybrid Latent Reasoning with Decoupled Policy Optimization

    cs.CV 2026-04 unverdicted novelty 7.0 of 10

    HyLaR with DePO enables effective RL in hybrid discrete-continuous spaces for multimodal models, outperforming prior MLLMs on perception and understanding benchmarks.

  6. Forest Before Trees: Latent Superposition for Efficient Visual Reasoning

    cs.CL 2026-01 unverdicted novelty 7.0 of 10

    Laser reformulates visual reasoning via Dynamic Windowed Alignment Learning to maintain latent superposition of global features, delivering 5.03% average gains over Monet and over 97% fewer inference tokens on six benchmarks.

  7. FinCriticalED: A Visual Benchmark for Financial Fact-Level OCR

    cs.CV 2025-11 unverdicted novelty 7.0 of 10

    FinCriticalED benchmark reveals that OCR and MLLM systems frequently fail to preserve critical financial facts such as numbers and monetary units even when lexical accuracy is high.

  8. OCRBench v2: An Improved Benchmark for Evaluating Large Multimodal Models on Visual Text Localization and Reasoning

    cs.CV 2024-12 accept novelty 7.0 of 10

    OCRBench v2 is a new benchmark with four times more tasks than prior versions that reveals most large multimodal models score below 50 out of 100 on visual text tasks and share five specific weaknesses.

  9. StochasT: Learning with Stochastic Turn Depth for Visual Instruction Tuning

    cs.CV 2026-07 unverdicted novelty 6.0 of 10

    StochasT uses stochastic clustering of language tasks into varying turn depths for the same image to improve LVLMs on both single-turn and multi-turn scenarios without discarding data.

  10. DataComp-VLM: Improved Open Datasets for Vision-Language Models

    cs.CV 2026-06 unverdicted novelty 6.0 of 10

    DataComp-VLM benchmark shows instruction-heavy data mixtures outperform caption-heavy ones for VLM training, with DCVLM-Baseline reaching 63.6% on 33 tasks using 200B tokens, +5.4pp over FineVision.

  11. ViTexQA: A Multi-Frame Temporal Perception Dataset for Video Text Question Answering

    cs.CV 2026-06 unverdicted novelty 6.0 of 10

    ViTexQA is a dataset forcing multi-frame text fusion for all questions, with FrameThinker achieving 6.3% ROUGE-L gain over baselines via CoT SFT and temporally-grounded RL.

  12. HPP: Hierarchical Programmatic Probing for Long Video Understanding by Decoupling Perception and Reasoning

    cs.CV 2026-06 unverdicted novelty 6.0 of 10

    HPP decouples perception from reasoning in long-video VLMs by having an LLM run iterative programmatic probes on hierarchically segmented video, reporting gains on LongVideoBench, EgoSchema, VideoMME, and MLVU.

  13. The Last Visible Pixel: Probing Fine-Scale Perception in Vision-Language Models

    cs.CV 2026-06 unverdicted novelty 6.0 of 10

    FineSightBench reveals VLMs perceive patterns down to 12px but show persistent failures in fine-scale reasoning such as numeracy and sequencing.

  14. From Failure to Feedback: Group Revision Unlocks Hard Cases in Object-Level Grounding

    cs.CV 2026-05 unverdicted novelty 6.0 of 10

    A group-revision paradigm for GRPO-based RL fine-tuning of VLMs converts failure responses into improvement signals that refine rewards and advantages, yielding gains on referring segmentation, REC, and counting benchmarks.

  15. Deep Pre-Alignment for VLMs

    cs.CV 2026-05 unverdicted novelty 6.0 of 10

    Deep Pre-Alignment uses a small VLM perceiver instead of ViT to pre-align visual features with LLM text space, yielding 1.9-3.0 point gains on multimodal benchmarks and 32.9% less language forgetting.

  16. DiM\textsuperscript{3}: Bridging Multilingual and Multimodal Models via Direction- and Magnitude-Aware Merging

    cs.CL 2026-05 conditional novelty 6.0 of 10

    DiM3 is a direction- and magnitude-aware merging method that composes heterogeneous multilingual and multimodal updates in LLM backbones, outperforming baselines on 57-language benchmarks while retaining multimodal pe...

  17. DiM\textsuperscript{3}: Bridging Multilingual and Multimodal Models via Direction- and Magnitude-Aware Merging

    cs.CL 2026-05 conditional novelty 6.0 of 10

    DiM3 merges multilingual and multimodal model updates in a direction- and magnitude-aware way to enhance multilingual performance in vision-language models while preserving original multimodal abilities.

  18. HyLaR: Hybrid Latent Reasoning with Decoupled Policy Optimization

    cs.CV 2026-04 conditional novelty 6.0 of 10

    HyLaR interleaves discrete text generation with continuous visual latent representations and optimizes them via a decoupled RL algorithm using vMF distributions, improving fine-grained visual reasoning.

  19. Visual Enhanced Depth Scaling for Multimodal Latent Reasoning

    cs.CV 2026-04 unverdicted novelty 6.0 of 10

    Visual replay module and adaptive depth scaling improve multimodal latent reasoning, reaching SOTA benchmarks with faster inference than explicit chain-of-thought methods.

  20. Omni-Diffusion: Unified Multimodal Understanding and Generation with Masked Discrete Diffusion

    cs.CV 2026-03 conditional novelty 6.0 of 10

    A unified mask-based discrete diffusion model jointly models text, speech, and image tokens and matches or exceeds several any-to-any and specialist multimodal baselines.

  21. Learning More from Less: Unlocking Internal Representations for Benchmark Compression

    cs.AI 2026-01 conditional novelty 6.0 of 10

    Using aligned hidden states from ten LLMs to select a small representative question set lets RepCore estimate full-benchmark accuracy and rankings more accurately than output-only methods.

  22. DeepEyesV2: Toward Agentic Multimodal Model

    cs.CV 2025-11 unverdicted novelty 6.0 of 10

    DeepEyesV2 uses a two-stage cold-start plus reinforcement learning pipeline to produce an agentic multimodal model that adaptively invokes tools and outperforms direct RL on real-world reasoning benchmarks.

  23. InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiency

    cs.CV 2025-08 unverdicted novelty 6.0 of 10

    InternVL3.5 advances open-source multimodal models with Cascade RL for +16% reasoning gains and ViR for 4x inference speedup, with the 241B model reaching SOTA among open-source MLLMs on multimodal, reasoning, and age...

  24. MM-BrowseComp: A Comprehensive Benchmark for Multimodal Browsing Agents

    cs.CL 2025-08 unverdicted novelty 6.0 of 10

    MM-BrowseComp is a multimodal browsing benchmark on which the best evaluated model, OpenAI o3 with tools, achieves 29.02% accuracy.

  25. ProactiveVideoQA: A Comprehensive Benchmark Evaluating Proactive Interactions in Video Large Language Models

    cs.CV 2025-07 conditional novelty 6.0 of 10

    ProactiveVideoQA is a benchmark for proactive video question answering, and the proposed PAUC metric jointly scores response timing and content, claiming better alignment with human preferences.

  26. InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Models

    cs.CV 2025-04 conditional novelty 6.0 of 10

    InternVL3-78B sets a new open-source SOTA of 72.2 on MMMU via native joint multimodal pre-training, V2PE, MPO, and test-time scaling while remaining competitive with proprietary models.

  27. Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling

    cs.CV 2024-12 unverdicted novelty 6.0 of 10

    InternVL 2.5 is the first open-source MLLM to surpass 70% on the MMMU benchmark via model, data, and test-time scaling, with a 3.7-point gain from chain-of-thought reasoning.

  28. ParVL: Parallel Scaling and Expandable Compute Allocation for Multimodal LLMs

    cs.CV 2026-08 conditional novelty 5.0 of 10

    ParVL scales MLLM computation by running multiple prefix-conditioned ViT and LLM branches over a shared backbone, improving average benchmark scores by 0.3 to 0.9 points and showing task-dependent vision-language allocation.

  29. Mage-VL: An Efficient Codec-Native Streaming Multimodal Foundation Model

    cs.CV 2026-07 conditional novelty 5.0 of 10

    Codec-guided sparse patch selection plus a lightweight speak/silent gate yields a 4B streaming VLM that is competitive on static tasks, stronger on video/spatial benchmarks, and much cheaper at inference.

  30. Aligning What Vision-Language Models See and Perceive with Adaptive Information Flow

    cs.CV 2026-04 unverdicted novelty 5.0 of 10

    An inference-time technique that uses token activation dynamics to adaptively restrict text attention to important visual tokens, improving VLM accuracy on VQA, grounding, counting, OCR, and hallucination benchmarks.

  31. Visual Enhanced Depth Scaling for Multimodal Latent Reasoning

    cs.CV 2026-04 unverdicted novelty 5.0 of 10

    Visual replay and depth scaling in latent reasoning produce state-of-the-art multimodal results with faster inference than explicit CoT.

  32. Visual Enhanced Depth Scaling for Multimodal Latent Reasoning

    cs.CV 2026-04 unverdicted novelty 5.0 of 10

    A visual replay module combined with adaptive depth scaling improves multimodal latent reasoning, delivering state-of-the-art benchmark results and faster inference than explicit chain-of-thought methods.

  33. Act Wisely: Cultivating Meta-Cognitive Tool Use in Agentic Multimodal Models

    cs.CV 2026-04 unverdicted novelty 5.0 of 10

    HDPO reframes tool efficiency as a conditional objective within accurate trajectories, enabling Metis to reduce tool invocations by orders of magnitude while raising reasoning accuracy.

  34. MathReal: We Keep It Real! A Real Scene Benchmark for Evaluating Math Reasoning in Multimodal Large Language Models

    cs.CV 2025-08 unverdicted novelty 5.0 of 10

    A new benchmark of 2,000 K-12 math questions photographed in real scenes finds that multimodal LLMs perform significantly worse on authentic handheld images than on clean benchmarks.

  35. BLUEX Revisited: Enhancing Benchmark Coverage with Automatic Captioning

    cs.CL 2025-08 conditional novelty 4.0 of 10

    An updated BLUEX benchmark with 1,422 questions and GPT-4o-generated captions that make image-based questions usable by text-only LLMs.

Pith tools