Pith. sign in

REVIEW 3 major objections 2 minor 256 cited by

SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehension

T0 review · 3 major / 2 minor · reviewed 2026-05-12 · grok-4.3

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read SEED-Bench supplies 19K human-verified multiple-choice questions to measure multimodal LLMs on image and video comprehension across 12 dimensions.

desk verdict SEED-Bench scales up multimodal LLM evaluation with a large MCQ set but risks not isolating visual comprehension. read the letter →

arxiv 2307.16125 v2 pith:2MD3PLBM submitted 2023-07-30 cs.CL cs.CV

Bohao Li , Rui Wang , Guangzhi Wang , Yuying Ge , Yixiao Ge , Ying Shan This is my paper · ORCID
classification cs.CLcs.CV
keywords multimodalLLMsgenerativecomprehensionbenchmarkimageunderstandingvideomultiplechoiceevaluationspatialandtemporalreasoning
checked against Foundation.DAlembert.InevitabilityFoundation.LogicAsFunctionalEquation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to create an objective way to test the generative comprehension skills of multimodal large language models. It builds a benchmark six times larger than prior ones, with questions that cover both static images and dynamic videos. A pipeline combines automatic generation with human checks to produce reliable multiple-choice items whose correct answers come directly from annotations. This setup allows models to be scored without needing extra human or model judges at evaluation time. Testing eighteen existing models then shows where current systems fall short in spatial and temporal understanding.

What carries the argument

The pipeline that generates multiple-choice questions targeting specific dimensions through automatic filtering followed by manual verification.

What would settle it

An experiment showing that models scoring highest on SEED-Bench still fail to produce accurate open-ended descriptions or answers on the same image and video content.

Watch

Extended reading notes

Core claim

SEED-Bench consists of 19K multiple choice questions with accurate human annotations, which spans 12 evaluation dimensions including the comprehension of both the image and video modality, enabling an objective and efficient assessment of model performance without human or GPT intervention during evaluation.

Load-bearing premise

The questions produced by automatic generation plus manual verification actually test genuine generative comprehension instead of artifacts from the creation process.

Editorial extensions

If this is right

  • Evaluating 18 models across all 12 dimensions reveals concrete limitations in current MLLMs for both spatial and temporal understanding.
  • The benchmark supports consistent leaderboard tracking that lets the community compare progress without repeated human judgment.
  • Insights from the results can directly motivate targeted improvements in models that handle image and video modalities together.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Widespread use of this benchmark could make cross-model comparisons more reliable by fixing the question set and scoring method.
  • The scale and verification process may encourage development of models that maintain performance when questions shift from multiple choice to free-form generation.
  • Extending similar pipelines to new modalities could help identify whether comprehension gaps are modality-specific or general.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 2 minor

Summary. The paper introduces SEED-Bench, a benchmark of 19K multiple-choice questions with human annotations for evaluating generative comprehension in Multimodal LLMs (MLLMs). It spans 12 dimensions covering spatial and temporal understanding of both image and video modalities, constructed via an automatic question-generation pipeline with filtering and manual verification. The authors evaluate 18 existing MLLMs on the benchmark, reveal their limitations, and announce a public leaderboard.

Significance. If validated to require genuine multimodal input, SEED-Bench would be a meaningful contribution due to its scale (six times larger than prior benchmarks) and broad coverage of 12 dimensions. A well-controlled benchmark of this size could standardize evaluation of MLLM comprehension and guide improvements in visual-language integration.

major comments (3)
  1. [Section 3] Benchmark construction (Section 3): The pipeline description provides no quantitative evidence that questions cannot be solved from question text and options alone (e.g., no text-only baseline accuracy reported, no ablation removing images/videos). This directly undermines the central claim that performance measures multimodal comprehension rather than language priors.
  2. [Section 3.2] Annotation process (Section 3.2): No inter-annotator agreement statistics or details on how the 12 evaluation dimensions were selected and operationalized are reported, weakening confidence that the 19K questions reliably target the intended spatial/temporal capabilities.
  3. [Section 4] Evaluation results (Section 4): The reported model scores lack analysis of whether errors correlate with visual content (e.g., via attention maps or controlled perturbations); without this, it is unclear whether the benchmark isolates the claimed generative comprehension limitations.
minor comments (2)
  1. [Abstract] The abstract and introduction repeat the 'x6 larger' claim without citing the exact sizes of the compared benchmarks.
  2. [Figure 1] Figure 1 caption could more explicitly label the 12 dimensions and their image/video split for quick reference.

Simulated Author's Rebuttal

3 responses · 0 unresolved

We thank the referee for the constructive and detailed comments, which help strengthen the manuscript. We address each major comment point by point below, agreeing where revisions are warranted and providing clarifications where the existing work already supports our claims. We will update the paper accordingly in the revised version.

read point-by-point responses

  1. Referee: [Section 3] Benchmark construction (Section 3): The pipeline description provides no quantitative evidence that questions cannot be solved from question text and options alone (e.g., no text-only baseline accuracy reported, no ablation removing images/videos). This directly undermines the central claim that performance measures multimodal comprehension rather than language priors.

    Authors: We agree that explicit quantitative validation is important to confirm the benchmark requires multimodal input. Although the questions are generated from visual content with human-annotated ground truth and filtered to target specific visual dimensions, we did not report a text-only baseline in the original submission. In the revised manuscript, we will add evaluations of multiple models on the text-only version of SEED-Bench, demonstrating substantially lower accuracy without images or videos. This will directly support that the benchmark measures generative multimodal comprehension rather than language priors alone. revision: yes

  2. Referee: [Section 3.2] Annotation process (Section 3.2): No inter-annotator agreement statistics or details on how the 12 evaluation dimensions were selected and operationalized are reported, weakening confidence that the 19K questions reliably target the intended spatial/temporal capabilities.

    Authors: We acknowledge the value of reporting inter-annotator agreement to increase confidence in the annotations. We will add these statistics (e.g., agreement rates across the manual verification step) to the revised Section 3.2. The 12 dimensions were selected to comprehensively cover spatial and temporal understanding for both images and videos, drawing from established categories in visual reasoning and video comprehension literature. We will expand the description of how each dimension is operationalized through targeted question templates and examples in the updated manuscript. revision: yes

  3. Referee: [Section 4] Evaluation results (Section 4): The reported model scores lack analysis of whether errors correlate with visual content (e.g., via attention maps or controlled perturbations); without this, it is unclear whether the benchmark isolates the claimed generative comprehension limitations.

    Authors: This is a fair point for deeper validation of error sources. The current results already show systematic weaknesses across models on specific dimensions (e.g., temporal reasoning), which we attribute to multimodal integration challenges based on the question design. However, attention map analysis or systematic perturbations would require additional experiments not included in this benchmark-focused work. In the revision, we will incorporate a qualitative error analysis with example cases linking failures to visual elements, along with a discussion of how such analyses could be pursued in future work. revision: partial

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: benchmark construction is descriptive and externally verifiable

full rationale

The paper introduces SEED-Bench via an explicit pipeline of automatic question generation, filtering, and human annotation/verification to produce 19K MCQs across 12 dimensions. No equations, fitted parameters, predictions, or derivations are claimed. The central claim (that the resulting questions enable objective evaluation of MLLM comprehension) rests on the described human-verified ground truth rather than reducing to self-definition or self-citation. Evaluation of 18 external models occurs after benchmark creation, providing an independent test. This matches the default expectation of a self-contained benchmark paper with no load-bearing circular steps.

Assumptions & free parameters 0 free parameters · 0 assumptions · 0 invented entities

This is an empirical benchmark paper with no mathematical derivations, fitted parameters, or new postulated entities; it relies on standard machine-learning evaluation practices and human annotation.

how reviews work

0 comments
Cite this review

Pith. "Pith review of SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehension." pith.science (2026). https://pith.science/paper/2MD3PLBM

@misc{pith2026230716125,
  author       = {Pith},
  title        = {Pith review of: SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehension},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2MD3PLBM}},
  note         = {Machine review of arXiv:2307.16125}
}
read the original abstract

Based on powerful Large Language Models (LLMs), recent generative Multimodal Large Language Models (MLLMs) have gained prominence as a pivotal research area, exhibiting remarkable capability for both comprehension and generation. In this work, we address the evaluation of generative comprehension in MLLMs as a preliminary step towards a comprehensive assessment of generative models, by introducing a benchmark named SEED-Bench. SEED-Bench consists of 19K multiple choice questions with accurate human annotations (x 6 larger than existing benchmarks), which spans 12 evaluation dimensions including the comprehension of both the image and video modality. We develop an advanced pipeline for generating multiple-choice questions that target specific evaluation dimensions, integrating both automatic filtering and manual verification processes. Multiple-choice questions with groundtruth options derived from human annotation enables an objective and efficient assessment of model performance, eliminating the need for human or GPT intervention during evaluation. We further evaluate the performance of 18 models across all 12 dimensions, covering both the spatial and temporal understanding. By revealing the limitations of existing MLLMs through evaluation results, we aim for SEED-Bench to provide insights for motivating future research. We will launch and consistently maintain a leaderboard to provide a platform for the community to assess and investigate model capability.

Discussion (0). Continue with ORCID to comment.

Forward citations

Showing 60 of 256 Pith papers that cite this

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. See all 256 Pith citations

  1. DataComp-VLM: Improved Open Datasets for Vision-Language Models

    cs.CV 2026-06 unverdicted novelty 8.0 of 10

    DataComp-VLM benchmark shows instruction-heavy data mixtures outperform caption-heavy ones for VLM training, with DCVLM-Baseline reaching 63.6% on 33 tasks using 200B tokens, +5.4pp over FineVision.

  2. MME-RealWorld: Could Your Multimodal LLM Challenge High-Resolution Real-World Scenarios that are Difficult for Humans?

    cs.CV 2024-08 conditional novelty 8.0 of 10

    MME-RealWorld is the largest manually annotated high-resolution benchmark for MLLMs, where even the best models achieve less than 60% accuracy on challenging real-world tasks.

  3. MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI

    cs.CL 2023-11 unverdicted novelty 8.0 of 10

    MMMU provides 11.5K heterogeneous college-level multimodal questions that current models solve at 56-59% accuracy, establishing a new standard for expert multimodal evaluation.

  4. Perception Before Supervision: Self-Contained Visual Distillation from Counterfactual Blind Spots

    cs.CV 2026-08 conditional novelty 7.0 of 10

    CVPD finds visual blind spots by comparing a model's own crop, full-image, and ghosted responses, then uses the crop as a positive teacher and the ghost as a negative teacher for dense self-distillation.

  5. Learning in Deep Networks under Dale's Constraint

    cs.AI 2026-08 reject novelty 7.0 of 10

    An on-off two-channel network with fixed-sign synapses and local Hebbian learning is claimed to recover backpropagation exactly under symmetric weights and to beat comparable vanilla networks on Tiny ImageNet.

  6. Contextualized Evaluation of Vision Language Models through Dynamic, Multi-turn Interactions

    cs.AI 2026-07 conditional novelty 7.0 of 10

    Context-guided multi-turn interviews reveal more VLM hallucinations than static benchmarks, and those hallucinations increase with conversational history and false-premise questions.

  7. On Test-Time Scaling for Vision-Language Models

    cs.CV 2026-06 unverdicted novelty 7.0 of 10

    Small well-performing LVLMs gain the most from test-time scaling with up to 30% improvements that can match or exceed larger models, while visual information is used mainly early in reasoning chains.

  8. TriViewBench: Controlled Complexity Scaling for Multi-View Structural Reasoning in MLLMs

    cs.CV 2026-06 unverdicted novelty 7.0 of 10

    TriViewBench shows all tested MLLMs follow the same capability order with sharp drops on complex multi-view tasks and near-zero gain from Chain-of-Thought prompting.

  9. Mind the Heads: Topological Representation Alignment for Multimodal LLMs

    cs.CV 2026-06 conditional novelty 7.0 of 10

    HeRA aligns least-aligned attention heads in MLLMs using an MKNN-based contrastive objective to preserve cross-modal topological structure, yielding gains on vision-centric tasks and reduced hallucinations across 18 b...

  10. NEST: Narrative Event Structures in Time for Long Video Understanding

    cs.CV 2026-06 unverdicted novelty 7.0 of 10

    NEST is a new benchmark dataset for narrative event structures in long videos, with baselines reporting ETD below 8%, EL under 6%, EAE below 11%, and ERE at 35-44% F1.

  11. GRIP: Feedback-Guided Prompt Retrieval for Large Multimodal Models

    cs.CV 2026-06 unverdicted novelty 7.0 of 10

    GRIP uses contrastive training on LMM feedback to retrieve beneficial in-context examples for multimodal tasks, outperforming similarity-based methods and transferring across models including GPT-4o.

  12. Reroute, Don't Remove: Recoverable Visual Token Routing for Vision-Language Models

    cs.CV 2026-06 conditional novelty 7.0 of 10

    Reroute turns irreversible visual-token pruning into recoverable routing that reuses existing attention scores, improving grounding performance under aggressive reduction on LLaVA-1.5 and Qwen while preserving TFLOPs ...

  13. CapRL++: Unified Reinforcement Learning with Verifiable Rewards for Dense Image and Video Captioning

    cs.CV 2026-06 unverdicted novelty 7.0 of 10

    CapRL++ applies reinforcement learning with verifiable rewards to dense image and video captioning by scoring captions via the accuracy of a vision-free LLM answering MCQs from the caption alone.

  14. Differentiable Efficient Operator Search

    cs.LG 2026-06 unverdicted novelty 7.0 of 10

    Introduces Efficient Operator Search, a differentiable framework that jointly optimizes token reduction locations, retention budgets, and operator behaviors in multimodal models under cost constraints, recovering manu...

  15. A Dataset for Dynamic Human Preferences for Vision Language Models

    cs.CV 2026-06 unverdicted novelty 7.0 of 10

    Introduces a benchmark dataset with automated pipeline for evaluating VLMs on dynamic in-context human preferences, distinct from static benchmarks.

  16. AVI-Bench: Toward Human-like Audio-Visual Intelligence of Omni-MLLMs

    cs.CV 2026-06 unverdicted novelty 7.0 of 10

    AVI-Bench is a cognitively inspired benchmark that evaluates Omni-MLLMs on joint audio-visual tasks and reveals substantial limitations in current models.

  17. EvoCut: Multi-Layer Evolution-Aware Visual Token Compression for Efficient Large Vision-Language Models

    cs.CV 2026-06 conditional novelty 7.0 of 10

    EvoCut is a training-free visual token compression technique that identifies important tokens via multi-layer evolution deviation, retaining 11.1% tokens with 94.4% average performance preserved on LLaVA-1.5-7B.

  18. VGenST-Bench: A Benchmark for Spatio-Temporal Reasoning via Active Video Synthesis

    cs.CV 2026-05 unverdicted novelty 7.0 of 10

    VGenST-Bench is a new video benchmark for MLLM spatio-temporal reasoning built via generative synthesis, a multi-agent pipeline with human oversight, a 3x2x2 taxonomy, and hierarchical tasks separating perception from...

  19. COHERENCE: Benchmarking Fine-Grained Image-Text Alignment in Interleaved Multimodal Contexts

    cs.CV 2026-04 unverdicted novelty 7.0 of 10

    COHERENCE is a new benchmark for measuring MLLMs' ability to recover fine-grained image-text correspondences in interleaved multimodal contexts.

  20. Seeing the Scene Matters: Revealing Forgetting in Video Understanding Models with a Scene-Aware Long-Video Benchmark

    cs.CV 2026-03 unverdicted novelty 7.0 of 10

    SceneBench shows VLMs sharply lose accuracy on scene-level long-video questions, and Scene-RAG only partially mitigates that forgetting (+2.50%).

  21. LLMind: Bio-inspired Training-free Adaptive Visual Representations for Vision-Language Models

    cs.CV 2026-03 unverdicted novelty 7.0 of 10

    LLMind uses bio-inspired non-uniform sampling via a Mobius module and closed-loop semantic feedback to retain 82-97% of full-resolution VLM performance with only 1-5% of pixels on VQA benchmarks.

  22. Mema: Memory-Augmented Adapter for Enhanced Vision-Language Understanding

    cs.CV 2026-02 unverdicted novelty 7.0 of 10

    Mema adds a stateful memory module to vision encoders that accumulates hierarchical visual features across layers and selectively injects portions back via feedback to preserve fine-grained cues, yielding consistent g...

  23. Do All Individual Layers Help? An Empirical Study of Task-Interfering Layers in Vision-Language Models

    cs.AI 2026-02 conditional novelty 7.0 of 10

    Empirical analysis identifies task-interfering layers in VLMs and proposes TaLo, a test-time method to bypass them for improved performance without training.

  24. VideoASMR-Bench: Can AI-Generated ASMR Videos Fool VLMs and Humans?

    cs.CV 2025-12 unverdicted novelty 7.0 of 10

    VideoASMR-Bench shows state-of-the-art VLMs fail to reliably detect AI-generated ASMR videos from real ones, though humans can still identify the fakes relatively easily.

  25. See, Hear, and Understand: Benchmarking Audiovisual Human Speech Understanding in Multimodal Large Language Models

    cs.CV 2025-12 unverdicted novelty 7.0 of 10

    AV-SpeakerBench is a new speaker-centered benchmark showing that top multimodal models still struggle with fine-grained audiovisual speech understanding, with Gemini 2.5 Pro leading but open models lagging on fusion.

  26. Discrete Guidance Matching: Exact Guidance for Discrete Flow Matching

    cs.LG 2025-09 conditional novelty 7.0 of 10

    Derives exact guidance transition rates for discrete flow matching models that require only one model evaluation per sampling step and unify prior approximation-based methods.

  27. Unveiling the Response of Large Vision-Language Models to Visually Absent Tokens

    cs.CV 2025-09 conditional novelty 7.0 of 10

    Feed-forward neurons in LVLMs encode whether a text token is visually grounded, and a detector built on these neurons can reduce hallucination by overriding or replacing ungrounded tokens.

  28. From Standalone LLMs to Integrated Intelligence: A Survey of Compound Al Systems

    cs.MA 2025-06 accept novelty 7.0 of 10

    A survey that defines Compound AI Systems, proposes a multi-dimensional taxonomy based on component roles and orchestration strategies, reviews four foundational paradigms, and identifies key challenges for future research.

  29. GuessBench: Sensemaking Multimodal Creativity in the Wild

    cs.CL 2025-06 conditional novelty 7.0 of 10

    A Minecraft-based benchmark shows vision-language models often fail to decode player-built creations, with accuracy falling sharply for rare concepts and low-resource languages.

  30. CoreMatching: A Co-adaptive Sparse Inference Framework with Token and Neuron Pruning for Comprehensive Acceleration of Vision-Language Models

    cs.LG 2025-05 conditional novelty 7.0 of 10

    CoreMatching couples token pruning and neuron pruning in vision-language models by selecting tokens that activate the most core neurons, achieving large inference speedups with minor accuracy loss.

  31. Nexus-Gen: Unified Image Understanding, Generation, and Editing via Prefilled Autoregression in Shared Embedding Space

    cs.CV 2025-04 conditional novelty 7.0 of 10

    Prefilled autoregression in a shared embedding space lets Nexus-Gen unify image understanding, generation, and editing, achieving competitive benchmark scores with a 7B model.

  32. Generative Multimodal Pretraining with Discrete Diffusion Timestep Tokens

    cs.CV 2025-04 conditional novelty 7.0 of 10

    Diffusion timestep tokens give large language models a recursive visual language that improves unified multimodal comprehension and generation relative to spatial patch tokens.

  33. PerceptionLM: Open-Access Data and Models for Detailed Visual Understanding

    cs.CV 2025-04 conditional novelty 7.0 of 10

    PerceptionLM releases 2.8M human-labeled fine-grained video QA pairs and spatio-temporal captions, plus models and a new benchmark, arguing that human data, not just synthetic data, is needed for detailed video understanding.

  34. FLARE: Fully Integration of Vision-Language Representations for Deep Cross-Modal Understanding

    cs.CV 2025-04 unverdicted novelty 7.0 of 10

    FLARE is a vision-language model family using text-guided vision encoding, context-aware alignment decoding, dual-semantic mapping loss, and text-driven VQA synthesis to achieve deep cross-modal integration, outperfor...

  35. DualToken: Towards Unifying Visual Understanding and Generation with Dual Visual Vocabularies

    cs.CV 2025-03 unverdicted novelty 7.0 of 10

    DualToken disentangles semantics and appearance via separate codebooks in one tokenizer, reporting 0.25 rFID, 82% ImageNet zero-shot accuracy, and gains over VILA-U on understanding and generation benchmarks.

  36. WorldSense: Evaluating Real-world Omnimodal Understanding for Multimodal LLMs

    cs.CV 2025-02 unverdicted novelty 7.0 of 10

    WorldSense provides the first benchmark requiring synergistic audio-video-text understanding on 1,662 real-world videos and 3,172 QA pairs, where the best current multimodal LLM reaches only 65.1% accuracy.

  37. Probing Visual Language Priors in VLMs

    cs.CV 2024-12 conditional novelty 7.0 of 10

    ViLP shows that vision-language models often answer from text priors instead of image content, and an image-corruption DPO method partially fixes this.

  38. HumanVBench: Probing Human-Centric Video Understanding in MLLMs with Automatically Synthesized Benchmarks

    cs.CV 2024-12 unverdicted novelty 7.0 of 10

    HumanVBench provides a 16-task benchmark for human-centric video understanding in MLLMs, created through automated annotation and distractor synthesis pipelines, and shows top models lag human performance on emotion p...

  39. IDEA-Bench: How Far are Generative Models from Professional Designing?

    cs.CV 2024-12 conditional novelty 7.0 of 10

    IDEA-Bench measures generative models on 100 professional design tasks and finds the best tested system scores only 22.48 out of 100.

  40. Leveraging Large Vision-Language Model as User Intent-aware Encoder for Composed Image Retrieval

    cs.IR 2024-12 conditional novelty 7.0 of 10

    CIR-LVLM fine-tunes Qwen-VL-Chat with LoRA and hybrid task and instance-specific prompts to produce query and target embeddings, achieving new state-of-the-art recall on Fashion-IQ, Shoes, and CIRR.

  41. AV-Odyssey Bench: Can Your Multimodal LLMs Really Understand Audio-Visual Information?

    cs.CV 2024-12 conditional novelty 7.0 of 10

    A new audio-visual benchmark shows current multimodal LLMs perform barely above random guessing, with audio perception errors as the dominant failure mode.

  42. Enhancing Few-Shot Vision-Language Classification with Large Multimodal Model Features

    cs.CV 2024-11 conditional novelty 7.0 of 10

    SAVs extract a sparse set of attention head outputs from a frozen large multimodal model and use them as nearest-centroid features, achieving state-of-the-art few-shot vision-language classification without finetuning.

  43. VidHal: Benchmarking Temporal Hallucinations in Vision LLMs

    cs.CV 2024-11 unverdicted novelty 7.0 of 10

    VidHal is a new benchmark that evaluates VLLM temporal hallucinations through a caption ordering task on videos with varying hallucination levels.

  44. PyramidDrop: Accelerating Your Large Vision-Language Models via Pyramid Visual Redundancy Reduction

    cs.CV 2024-10 accept novelty 7.0 of 10

    PyramidDrop accelerates LVLMs by staged, similarity-based dropping of visual tokens that become redundant in deeper layers, delivering 40% faster training and 55% lower inference cost with comparable accuracy.

  45. Janus: Decoupling Visual Encoding for Unified Multimodal Understanding and Generation

    cs.CV 2024-10 unverdicted novelty 7.0 of 10

    Janus decouples visual encoding into task-specific pathways inside a single autoregressive transformer to unify multimodal understanding and generation while outperforming earlier unified models.

  46. We-Math: Does Your Large Multimodal Model Achieve Human-like Mathematical Reasoning?

    cs.AI 2024-07 accept novelty 7.0 of 10

    WE-MATH benchmark reveals most LMMs rely on rote memorization for visual math while GPT-4o has shifted toward knowledge generalization.

  47. MLVU: Benchmarking Multi-task Long Video Understanding

    cs.CV 2024-06 conditional novelty 7.0 of 10

    MLVU is a new benchmark for long video understanding that uses extended videos across diverse genres and multi-task evaluations, revealing that current MLLMs struggle significantly and degrade sharply with longer durations.

  48. Cracking the Code of Juxtaposition: Can AI Models Understand the Humorous Contradictions

    cs.CL 2024-05 unverdicted novelty 7.0 of 10

    Introduces YesBut benchmark showing state-of-the-art multimodal models lag humans on interpreting humorous contradictions in comics.

  49. MathVerse: Does Your Multi-modal LLM Truly See the Diagrams in Visual Math Problems?

    cs.CV 2024-03 conditional novelty 7.0 of 10

    MathVerse is a benchmark that tests multi-modal LLMs on visual math by providing each problem in six versions with progressively less diagram and text information to measure true visual understanding.

  50. HallusionBench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision-Language Models

    cs.CV 2023-10 unverdicted novelty 7.0 of 10

    HallusionBench shows GPT-4V reaches only 31.42% accuracy on paired questions testing language hallucination and visual illusion in LVLMs, with other models below 16%.

  51. Diagnosing Long-Video Quantitative Reasoning in Multimodal LLMs via Enumeration and Counting

    cs.CV 2026-03 accept novelty 6.5 of 10

    EC-Bench finds best MLLMs score 29.98% enumeration F1 and 23.74% counting accuracy on 152 hour-scale videos, with errors driven by instance identification and temporal grounding rather than arithmetic.

  52. DistMoE: Private-data Rehearsal-free Routing in Mixture-of-Experts for Distributed Instruction Tuning

    cs.CV 2026-08 conditional novelty 6.0 of 10

    DistMoE composes independently trained per-client Mixture-of-Experts specialists for vision-language models using a shared public anchor and isotropic residual calibration, without rehearsal of other clients' private data.

  53. LLMRouter: Unified Infrastructure for Developing, Evaluating, and Deploying LLM Routers

    cs.CL 2026-08 conditional novelty 6.0 of 10

    A new open-source library and benchmark, xRouteBench, evaluates LLM routers on a shared cost-aware protocol across text, memory, vision, time-series, and personalized tasks.

  54. TruthLens: Object Hallucination Detection via Self-Evaluating Truthfulness Scores in LVLMs

    cs.CV 2026-08 conditional novelty 6.0 of 10

    TruthLens fine-tunes LVLMs so the log-probability of a special token at each object mention becomes a truthfulness score, detecting object hallucinations with state-of-the-art AUROC.

  55. Disentangling 3D Modeling from Spatial Reasoning

    cs.LG 2026-08 conditional novelty 6.0 of 10

    DiSR splits spatial reasoning into 3D perception by frozen expert models and language-based reasoning by a LoRA-tuned LLM, reporting state-of-the-art results on 3DSRBench and SPAR-Bench with 0.33M training samples.

  56. Not All Redundant Tokens Are Alike: Analyzing Visual Token Pruning through Token Roles

    cs.CV 2026-08 conditional novelty 6.0 of 10

    Visual token pruning methods show distinct token-role biases, yet these biases do not correlate with downstream performance, and protecting dead tokens can sometimes maintain or improve accuracy.

  57. Learning to Predict Middle-Layer Attention in MLLMs for Visual Token Prunin

    cs.AI 2026-08 conditional novelty 6.0 of 10

    MAP predicts sample-specific middle-layer attention with a lightweight model and prunes visual tokens before the LLM, keeping 97.5% of performance at 5.56% of tokens.

  58. dRAE: Representation Autoencoder with Hyper-Spherical Codes

    cs.CV 2026-07 conditional novelty 6.0 of 10

    Switching codebook assignment and update to cosine similarity while keeping a magnitude-preserving commitment loss avoids codebook collapse and scales visual tokenizers to 131,072 codes with high utilization.

  59. Can Multimodal Large Language Models Understand OCT?

    cs.CV 2026-07 conditional novelty 6.0 of 10

    OCT-Bench, a 20-task benchmark across 10,076 questions, shows current MLLMs score up to 62% overall but only 43% on clinical reasoning over OCT images.

  60. Groc-PO: Grounded Context Preference Optimization for Truthful Multimodal LLMs

    cs.CV 2026-07 conditional novelty 6.0 of 10

    Groc-PO applies preference optimization at three grounded stages — object grounding, contextual grounding, grounded reasoning — and outperforms final-answer-only DPO on hallucination and complex-reasoning benchmarks.

See all 256 Pith citations

Reference graph

Works this paper leans on

42 extracted references · 42 canonical work pages · cited by 256 Pith papers (see all)

  1. [1]

    Scaling Instruction-Finetuned Language Models

    Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. Scaling instruction-finetuned language models. arXiv preprint arXiv:2210.11416, 2022

    work page Pith review arXiv 2022

  2. [2]

    Gpt-4 technical report, 2023

    OpenAI. Gpt-4 technical report, 2023

    work page 2023

  3. [3]

    Introducing chatgpt

    OpenAI. Introducing chatgpt. https://openai.com/blog/chatgpt, 2022

    work page 2022

  4. [4]

    FastChat. Vicuna. https://github.com/lm-sys/FastChat, 2023

    work page 2023

  5. [5]

    LLaMA: Open and Efficient Foundation Language Models

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023

    work page Pith review arXiv 2023

  6. [6]

    Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. ICML, 2023

    work page 2023

  7. [7]

    MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models

    Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304.10592, 2023

    work page Pith review arXiv 2023

  8. [8]

    Visual Instruction Tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. arXiv preprint arXiv:2304.08485, 2023

    work page Pith review arXiv 2023

Show all 42 references
  1. [9]

    mplug-owl: Modularization empowers large language models with multimodality

    Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang, Anwen Hu, Pengcheng Shi, Yaya Shi, et al. mplug-owl: Modularization empowers large language models with multimodality. arXiv preprint arXiv:2304.14178, 2023

    2023

  2. [10]

    Instructblip: Towards general-purpose vision-language models with instruction tuning

    Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. Instructblip: Towards general-purpose vision-language models with instruction tuning. arXiv preprint arXiv:2305.06500, 2023

    2023

  3. [11]

    Otter: A multi-modal model with in-context instruction tuning

    Bo Li, Yuanhan Zhang, Liangyu Chen, Jinghao Wang, Jingkang Yang, and Ziwei Liu. Otter: A multi-modal model with in-context instruction tuning. arXiv preprint arXiv:2305.03726, 2023

    2023

  4. [12]

    Multimodal-gpt: A vision and language model for dialogue with humans, 2023

    Tao Gong, Chengqi Lyu, Shilong Zhang, Yudong Wang, Miao Zheng, Qian Zhao, Kuikun Liu, Wenwei Zhang, Ping Luo, and Kai Chen. Multimodal-gpt: A vision and language model for dialogue with humans, 2023

    2023

  5. [13]

    Pandagpt: One model to instruction- follow them all

    Yixuan Su, Tian Lan, Huayang Li, Jialu Xu, Yan Wang, and Deng Cai. Pandagpt: One model to instruction- follow them all. arXiv preprint arXiv:2305.16355, 2023

    2023

  6. [14]

    Kosmos-2: Grounding multimodal large language models to the world

    Zhiliang Peng, Wenhui Wang, Li Dong, Yaru Hao, Shaohan Huang, Shuming Ma, and Furu Wei. Kosmos-2: Grounding multimodal large language models to the world. arXiv preprint arXiv:2306.14824, 2023

    2023 arXiv

  7. [15]

    Videochat: Chat-centric video understanding

    KunChang Li, Yinan He, Yi Wang, Yizhuo Li, Wenhai Wang, Ping Luo, Yali Wang, Limin Wang, and Yu Qiao. Videochat: Chat-centric video understanding. arXiv preprint arXiv:2305.06355, 2023

    2023

  8. [16]

    Video-chatgpt: Towards detailed video understanding via large vision and language models

    Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Shahbaz Khan. Video-chatgpt: Towards detailed video understanding via large vision and language models. arXiv preprint arXiv:2306.05424 , 2023

    2023

  9. [17]

    Valley: Video assistant with large language model enhanced ability.arXiv preprint arXiv:2306.07207, 2023

    Ruipu Luo, Ziwang Zhao, Min Yang, Junwei Dong, Minghui Qiu, Pengcheng Lu, Tao Wang, and Zhongyu Wei. Valley: Video assistant with large language model enhanced ability.arXiv preprint arXiv:2306.07207, 2023

    2023

  10. [18]

    Planting a seed of vision in large language model

    Yuying Ge, Yixiao Ge, Ziyun Zeng, Xintao Wang, and Ying Shan. Planting a seed of vision in large language model. arXiv preprint arXiv:2307.08041, 2023

    2023

  11. [19]

    Generative pretraining in multimodality

    Quan Sun, Qiying Yu, Yufeng Cui, Fan Zhang, Xiaosong Zhang, Yueze Wang, Hongcheng Gao, Jingjing Liu, Tiejun Huang, and Xinlong Wang. Generative pretraining in multimodality. arXiv preprint arXiv:2307.05222, 2023

    2023

  12. [20]

    Scaling autoregressive multi-modal models: Pretraining and instruction tuning

    Yu Lili, Shi Bowen, Pasunuru Ram, Miller Benjamin, Golovneva Olga, Wang Tianlu, Babu Arun, Tang Binh, Karrer Brian, Sheynin Shelly, Ross Candace, Polyak Adam, Howes Russ, Sharma Vasu, Xu Jacob, Singer Uriel, Li (AI) Daniel, Ghosh Gargi, Taigman Yaniv, Fazel-Zarandi Maryam, Cel...

    2023

  13. [21]

    Generating images with multimodal language models

    Jing Yu Koh, Daniel Fried, and Ruslan Salakhutdinov. Generating images with multimodal language models. arXiv preprint arXiv:2305.17216, 2023

    2023

  14. [22]

    Making the v in vqa matter: Elevating the role of image understanding in visual question answering

    Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 6904–6913, 2017. 13

    2017

  15. [23]

    Mme: A comprehensive evaluation benchmark for multimodal large language models

    Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Zhenyu Qiu, Wei Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, and Rongrong Ji. Mme: A comprehensive evaluation benchmark for multimodal large language models. arXiv preprint arXiv:2306.13394, 2023

    2023 arXiv

  16. [24]

    Lamm: Language-assisted multi-modal instruction-tuning dataset, framework, and benchmark

    Zhenfei Yin, Jiong Wang, Jianjian Cao, Zhelun Shi, Dingning Liu, Mukai Li, Lu Sheng, Lei Bai, Xiaoshui Huang, Zhiyong Wang, et al. Lamm: Language-assisted multi-modal instruction-tuning dataset, framework, and benchmark. arXiv preprint arXiv:2306.06687, 2023

    2023

  17. [25]

    Lvlm-ehub: A comprehensive evaluation benchmark for large vision-language models

    Peng Xu, Wenqi Shao, Kaipeng Zhang, Peng Gao, Shuo Liu, Meng Lei, Fanqing Meng, Siyuan Huang, Yu Qiao, and Ping Luo. Lvlm-ehub: A comprehensive evaluation benchmark for large vision-language models. arXiv preprint arXiv:2306.09265, 2023

    2023

  18. [26]

    Mmbench: Is your multi-modal model an all-around player? arXiv preprint arXiv:2307.06281, 2023

    Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. Mmbench: Is your multi-modal model an all-around player? arXiv preprint arXiv:2307.06281, 2023

    2023

  19. [27]

    Tag2text: Guiding vision-language model via image tagging

    Xinyu Huang, Youcai Zhang, Jinyu Ma, Weiwei Tian, Rui Feng, Yuejie Zhang, Yaqian Li, Yandong Guo, and Lei Zhang. Tag2text: Guiding vision-language model via image tagging. arXiv preprint arXiv:2303.05657, 2023

    2023

  20. [28]

    Grit: A generative region-to-text transformer for object understanding

    Jialian Wu, Jianfeng Wang, Zhengyuan Yang, Zhe Gan, Zicheng Liu, Junsong Yuan, and Lijuan Wang. Grit: A generative region-to-text transformer for object understanding. arXiv preprint arXiv:2212.00280, 2022

    2022

  21. [29]

    Berg, Wan-Yen Lo, Piotr Dollár, and Ross Girshick

    Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo, Piotr Dollár, and Ross Girshick. Segment anything. arXiv:2304.02643, 2023

    2023 arXiv

  22. [30]

    Vinvl: Revisiting visual representations in vision-language models

    Pengchuan Zhang, Xiujun Li, Xiaowei Hu, Jianwei Yang, Lei Zhang, Lijuan Wang, Yejin Choi, and Jianfeng Gao. Vinvl: Revisiting visual representations in vision-language models. In CVPR, 2021

    2021

  23. [31]

    Paddleocr

    https://github.com/PaddlePaddle/PaddleOCR. Paddleocr

  24. [32]

    Language models are few-shot learners

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems , 33:1877–1901, 2020

    1901

  25. [33]

    What makes for good visual tokenizers for large language models? arXiv preprint arXiv:2305.12223, 2023

    Guangzhi Wang, Yixiao Ge, Xiaohan Ding, Mohan Kankanhalli, and Ying Shan. What makes for good visual tokenizers for large language models? arXiv preprint arXiv:2305.12223, 2023

    2023

  26. [34]

    Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning

    Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In ACL, 2018

    2018

  27. [35]

    something something

    Raghav Goyal, Samira Ebrahimi Kahou, Vincent Michalski, Joanna Materzynska, Susanne Westphal, Heuna Kim, Valentin Haenel, Ingo Fruend, Peter Yianilos, Moritz Mueller-Freitag, et al. The" something something" video database for learning and evaluating visual common sense. In ICCV, 2017

    2017

  28. [36]

    Rescaling egocentric vision

    Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Antonino Furnari, Evangelos Kazakos, Jian Ma, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, et al. Rescaling egocentric vision. arXiv preprint arXiv:2006.13256, 2020

    2006

  29. [37]

    The language of actions: Recovering the syntax and semantics of goal-directed human activities

    Hilde Kuehne, Ali Arslan, and Thomas Serre. The language of actions: Recovering the syntax and semantics of goal-directed human activities. In CVPR, 2014

    2014

  30. [38]

    Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation

    Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In ICML, 2022

    2022

  31. [39]

    Truthfulqa: Measuring how models mimic human falsehoods

    Stephanie Lin, Jacob Hilton, and Owain Evans. Truthfulqa: Measuring how models mimic human falsehoods. arXiv preprint arXiv:2109.07958, 2021

    2021 arXiv

  32. [40]

    Transfer visual prompt generator across llms

    Ao Zhang, Hao Fei, Yuan Yao, Wei Ji, Li Li, Zhiyuan Liu, and Tat-Seng Chua. Transfer visual prompt generator across llms. abs/23045.01278, 2023

    2023

  33. [41]

    Openflamingo

    ml_foundations. Openflamingo. https://github.com/mlfoundations/open_flamingo, 2023

    2023

  34. [42]

    Llama-adapter v2: Parameter-efficient visual instruction model

    Peng Gao, Jiaming Han, Renrui Zhang, Ziyi Lin, Shijie Geng, Aojun Zhou, Wei Zhang, Pan Lu, Conghui He, Xiangyu Yue, Hongsheng Li, and Yu Qiao. Llama-adapter v2: Parameter-efficient visual instruction model. arXiv preprint arXiv:2304.15010, 2023. 14

    2023

Pith tools

Reviewed May 12, 2026 · model on record in the stance chip above.