Pith. sign in

REVIEW 60 cited by

Ovis: Structural Embedding Alignment for Multimodal Large Language Model

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2405.20797 v2 pith:Q7ENI4Q5 submitted 2024-05-31 cs.CV cs.AIcs.CLcs.LG

Shiyin Lu , Yang Li , Qing-Guo Chen , Zhao Xu , Weihua Luo , Kaifu Zhang , Han-Jia Ye This is my paper · ORCID
classification cs.CVcs.AIcs.CLcs.LG
keywords visualembeddingovisembeddingsmultimodaltextualmllmsstructural
verification ladder T0 review T1 audit T2 compute T3 formal

Signed reviews

No signed human review yet.

0 comments
read the original abstract

Current Multimodal Large Language Models (MLLMs) typically integrate a pre-trained LLM with another pre-trained vision transformer through a connector, such as an MLP, endowing the LLM with visual capabilities. However, the misalignment between two embedding strategies in MLLMs -- the structural textual embeddings based on an embedding look-up table and the continuous embeddings generated directly by the vision encoder -- makes challenges for a more seamless fusion of visual and textual information. We propose Ovis, a novel MLLM architecture designed to structurally align visual and textual embeddings. Ovis integrates an additional learnable visual embedding table into the visual encoder's process. To capture rich visual semantics, each image patch indexes the visual embedding table multiple times, resulting in a final visual embedding that is a probabilistic combination of the indexed embeddings. This structural approach mirrors the method used for generating textual embeddings. Empirical evaluations on various multimodal benchmarks show that Ovis outperforms open-source MLLMs of similar parameter scales and even surpasses the proprietary model Qwen-VL-Plus overall. These results highlight the potential of Ovis' structured visual representation for advancing MLLM architectural design and promoting more effective multimodal learning. Code, datasets, and models are available at https://github.com/AIDC-AI/Ovis.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 60 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. VISTA-Bench: Do Vision-Language Models Really Understand Visualized Text as Well as Pure Text?

    cs.CV 2026-02 conditional novelty 7.0 of 10

    VISTA-Bench shows vision-language models degrade on visualized text in images compared to equivalent pure text, with larger gaps under increased perceptual difficulty.

  2. Movie Facts and Fibs (MF$^2$): A Benchmark for Long Movie Understanding

    cs.CV 2025-06 conditional novelty 7.0 of 10

    MF2 evaluates long-movie understanding by asking models to classify fact/fib claim pairs; the best model trails humans by 23.5 points in pairwise accuracy.

  3. Seeing from Another Perspective: Evaluating Multi-View Understanding in MLLMs

    cs.CV 2025-04 conditional novelty 7.0 of 10

    All-Angles Bench, a 2,132-question benchmark across 90 real scenes, shows current MLLMs score around 60% on multi-view understanding while humans score 82%, with the largest gaps in camera pose estimation and cross-vi...

  4. Enhancing Few-Shot Vision-Language Classification with Large Multimodal Model Features

    cs.CV 2024-11 conditional novelty 7.0 of 10

    SAVs extract a sparse set of attention head outputs from a frozen large multimodal model and use them as nearest-centroid features, achieving state-of-the-art few-shot vision-language classification without finetuning.

  5. LookBack: Where and How to Score LVLM Responses via Visual Reference Usage

    cs.CV 2026-08 conditional novelty 6.0 of 10

    LookBack scores LVLM responses by calibrating token likelihood with an attention-based visual lookback score and weighting by visual relevance, improving Best-of-N selection over baselines.

  6. VLZip: Unified Visual and Textual Compression for Interleaved Long-Context Modeling

    cs.CV 2026-08 conditional novelty 6.0 of 10

    VLZip compresses visual and textual chunks into layer-specific soft prefixes, enabling a 3B model to train on 120K tokens and reason beyond 280K tokens, with top scores on a new narrative benchmark.

  7. CritiqueDriveVLM: From Verifier-Guided Reinforcement Learning to Latent Thought Distillation for Autonomous Driving

    cs.CV 2026-07 conditional novelty 6.0 of 10

    Verifier-guided multi-turn RL plus latent-state distillation raises DriveLMM-o1 MCQ to 76.54% for a tool-free teacher and yields a 28-token student at 416 ms (88% faster).

  8. BVS: Bayesian Visual Search with Multimodal Large Language Model for Fine-grained Perception

    cs.CV 2026-07 conditional novelty 6.0 of 10

    BVS combines early-stop attention rollout priors with a scale-aware non-stationary kernel and GP-UCB to locate tiny objects in UHR images more accurately and with fewer MLLM queries than prior visual-search methods.

  9. TSHA: A Benchmark for Visual Language Models in Trustworthy Safety Hazard Assessment Scenarios

    cs.CV 2026-03 unverdicted novelty 6.0 of 10

    TSHA is a new 80,000-pair benchmark for indoor safety hazard assessment; current vision-language models score roughly 45-85, and fine-tuning on TSHA raised Qwen2.5-VL-3B by 18.3 points on TSHA's test set.

  10. RefBench-PRO: Perceptual and Reasoning Oriented Benchmark for Referring Expression Comprehension

    cs.CV 2025-12 conditional novelty 6.0 of 10

    RefBench-PRO organizes REC into attribute, position, interaction, relation, commonsense, and reject tasks; no tested MLLM exceeds 72%, and Ref-R1 raises Qwen2.5-VL-7B from 57.6 to 69.4 on it.

  11. Diagnosing Corruption-Induced Reliability Failures in Vision-Language Models

    cs.CV 2025-11 conditional novelty 6.0 of 10

    Mild visual corruption can boost a vision-language model's top-1 accuracy while its confidence–correctness alignment (measured by the new RAS score) degrades.

  12. Measuring Epistemic Humility in Multimodal Large Language Models

    cs.CV 2025-09 conditional novelty 6.0 of 10

    A new 22,831-question visual benchmark shows that major multimodal LLMs struggle to reject false answer options, often scoring near random when abstaining is the only correct response.

  13. Towards Better Dental AI: A Multimodal Benchmark and Instruction Dataset for Panoramic X-ray Analysis

    cs.CV 2025-09 reject novelty 6.0 of 10

    MMOral is a large new dental X-ray instruction dataset and benchmark, but the proposed model's 24.73% improvement is from fine-tuning and then testing on the same data pool.

  14. KRETA: A Benchmark for Korean Reading and Reasoning in Text-Rich VQA Attuned to Diverse Visual Contexts

    cs.CV 2025-08 conditional novelty 6.0 of 10

    KRETA, a 2,577-item Korean text-rich VQA benchmark, shows vision-language models recognize Korean text well but lag in multi-step reasoning, especially in open-source models.

  15. Vision-G1: Towards General Vision Language Reasoning with Multi-Domain Data Curation

    cs.CV 2025-08 reject novelty 6.0 of 10

    Vision-G1 trains a 7B VLM with multi-round GRPO on an influence-filtered, difficulty-curated 46-source dataset spanning eight visual reasoning domains.

  16. In-context Learning of Vision Language Models for Detection of Physical and Digital Attacks against Face Recognition Systems

    cs.CV 2025-07 conditional novelty 6.0 of 10

    An in-context learning framework with open-source vision-language models detects face presentation and morphing attacks without training, beating CLIP-based zero-shot baselines on PAD but with performance highly sensi...

  17. UniCode$^2$: Cascaded Large-scale Codebooks for Unified Multimodal Understanding and Generation

    cs.CV 2025-06 conditional novelty 6.0 of 10

    UniCode² builds a 500K-entry codebook from clustered SigLIP embeddings and uses a cascaded frozen-plus-trainable codebook to unify multimodal understanding and generation with stable training and high token utilization.

  18. GenRecal: Generation after Recalibration from Large to Small Vision-Language Models

    cs.CL 2025-06 conditional novelty 6.0 of 10

    A learnable Recalibrator bridges different tokenizers so that small VLMs can distill knowledge from any large VLM, improving their benchmark scores.

  19. Text-Aware Image Restoration with Diffusion Models

    cs.CV 2025-06 conditional novelty 6.0 of 10

    A diffusion restoration model jointly trained with a text-spotting module and prompted by its own recognized text improves text recognition accuracy on restored images compared with general-purpose restoration methods.

  20. WebUIBench: A Comprehensive Benchmark for Evaluating Multimodal Large Language Models in WebUI-to-Code

    cs.CL 2025-06 conditional novelty 6.0 of 10

    WebUIBench is a 21,793-question benchmark that splits WebUI-to-Code into perception, HTML programming, and cross-modal understanding, and it ranks 29 multimodal LLMs on each sub-skill.

  21. SparseMM: Head Sparsity Emerges from Visual Concept Responses in MLLMs

    cs.CV 2025-06 conditional novelty 6.0 of 10

    SparseMM shows that under 5% of attention heads carry visual understanding in MLLMs and uses OCR-derived head scores to allocate KV-cache budgets asymmetrically, preserving accuracy at low cache sizes.

  22. GThinker: Towards General Multimodal Reasoning via Cue-Guided Rethinking

    cs.CV 2025-06 conditional novelty 6.0 of 10

    A cue-tagging and rethinking training recipe lifts a 7B multimodal model to 81.5% on M3CoT, though the gain may reflect training on the same benchmark.

  23. Affordance Benchmark for MLLMs

    cs.CL 2025-06 conditional novelty 6.0 of 10

    A new 2,000-question benchmark finds multimodal AI models recognize object affordances far worse than humans, with top model Gemini-2.0-Pro at 18.05% versus 85.34% human best.

  24. MedBookVQA: A Systematic and Comprehensive Medical Benchmark Derived from Open-Access Book

    cs.AI 2025-06 conditional novelty 6.0 of 10

    MedBookVQA is a new 5,000-question, textbook-derived multimodal benchmark for testing medical AI systems, with labels for imaging modality, body anatomy, and clinical specialty.

  25. mRAG: Elucidating the Design Space of Multi-modal Retrieval-Augmented Generation

    cs.AI 2025-05 conditional novelty 6.0 of 10

    A systematic empirical study finds that for multimodal RAG, EVA-CLIP retrieval, listwise LVLM reranking, and feeding only the top-ranked document works best, with a self-reflection agent adding further gains.

  26. AutoJudger: An Agent-Driven Framework for Efficient Benchmarking of MLLMs

    cs.CL 2025-05 conditional novelty 6.0 of 10

    An agent-driven framework adaptively selects a small subset of benchmark questions for MLLMs, preserving over 90% ranking accuracy with roughly 4-5% of the data.

  27. WildDoc: How Far Are We from Achieving Comprehensive and Robust Document Understanding in the Wild?

    cs.CV 2025-05 conditional novelty 6.0 of 10

    The paper introduces WildDoc, a photo-based document understanding benchmark, and shows that current vision-language models drop sharply in both accuracy and consistency on real-world captures.

  28. RePOPE: Impact of Annotation Errors on the POPE Benchmark

    cs.CV 2025-04 conditional novelty 6.0 of 10

    RePOPE corrects annotation errors in the POPE benchmark and shows that these errors cause significant shifts in model rankings, especially in F1 scores.

  29. VisuLogic: A Benchmark for Evaluating Visual Reasoning in Multi-modal Large Language Models

    cs.CV 2025-04 conditional novelty 6.0 of 10

    On VisuLogic's 1,000 vision-centric puzzles, the best multimodal models reach 28.1% accuracy versus a 24.9% random baseline and 51.4% human accuracy, and an RL baseline lifts accuracy by up to 5.6 points.

  30. MobileA3gent: Training Mobile GUI Agents Using Decentralized Self-Sourced Data from Diverse Users

    cs.AI 2025-02 conditional novelty 6.0 of 10

    A hierarchical auto-annotation pipeline plus episode-aware federated aggregation lets mobile GUI agents be trained on automatically labeled user trajectories at about 1% of human annotation cost.

  31. Poison as Cure: Visual Noise for Mitigating Object Hallucinations in LVMs

    cs.CV 2025-01 conditional novelty 6.0 of 10

    An inference-time visual perturbation, optimized with zeroth-order gradients and CLIP text similarity, modestly reduces object hallucination across 8 LVMs on POPE, BEAF, and CHAIR.

  32. Global Semantic-Guided Sub-image Feature Weight Allocation in High-Resolution Large Vision-Language Models

    cs.CV 2025-01 conditional novelty 6.0 of 10

    A semantic-similarity weighting module for sub-images improves high-resolution vision-language model performance, pending clarification of whether evaluation benchmarks overlap with training data.

  33. Compositional Generative Model of Unbounded 4D Cities

    cs.CV 2025-01 conditional novelty 6.0 of 10

    CityDreamer4D is a compositional generative model that creates unbounded, temporally coherent 4D cities by separately generating static scenes, buildings, and vehicles with neural fields.

  34. Are They the Same? Exploring Visual Correspondence Shortcomings of Multimodal LLMs

    cs.CV 2025-01 conditional novelty 6.0 of 10

    Multimodal LLMs show systematic weaknesses in instance-level visual correspondence, and CoLVA, trained with a fine-grained vision expert and object-level contrastive learning, reaches 49.8% accuracy on the new MMVM be...

  35. LUSIFER: Language Universal Space Integration for Enhanced Multilingual Embeddings with Large Language Models

    cs.CL 2025-01 conditional novelty 6.0 of 10

    A two-stage, English-only training method aligns XLM-R representations with Mistral-7B to produce multilingual embeddings, improving low-resource language scores by up to 22 points while losing about 10 points on English.

  36. Chimera: Improving Generalist Model with Domain-Specific Experts

    cs.CV 2024-12 conditional novelty 6.0 of 10

    Chimera fuses frozen domain-expert encoders into a generalist multimodal LLM via routing and a 30% masking of general tokens, lifting InternVL2-8B from 61.6 to 64.9 on MathVista and from 31.3 to 32.4 on MathVerse.

  37. Evaluating and Advancing Multimodal Large Language Models in Perception Ability Lens

    cs.CV 2024-11 conditional novelty 6.0 of 10

    AbilityLens unifies 11 public benchmarks into six perception abilities with accuracy and stability metrics, and reveals ability conflicts during MLLM training linked to data mixing and model size.

  38. Insight-V: Exploring Long-Chain Visual Reasoning with Multimodal Large Language Models

    cs.CV 2024-11 conditional novelty 6.0 of 10

    Insight-V uses machine-generated long-chain reasoning data, a reasoning-plus-summary multi-agent setup, and iterative DPO to improve visual reasoning scores of multimodal LLMs.

  39. VAGUE: Visual Contexts Clarify Ambiguous Expressions

    cs.CV 2024-11 conditional novelty 6.0 of 10

    A new 1,677-item benchmark shows that vision-language models struggle to resolve ambiguous indirect expressions even when they are given the visual context that makes the intent clear.

  40. MMGenBench: Fully Automatically Evaluating LMMs from the Text-to-Image Generation Perspective

    cs.CV 2024-11 conditional novelty 6.0 of 10

    A fully automatic benchmark pipeline grades LMMs by regenerating images from their captions and measuring embedding similarity to the originals, ranking over 50 models.

  41. NaviDC-OCR: Navigating Document Parsing Across Digital and Camera-Captured Documents

    cs.CV 2026-08 conditional novelty 5.0 of 10

    NaviDC-OCR is a 1.2B parameter document parser that combines deformation-aware training, adaptive layout sampling, and content-structure decoupled learning to reach state-of-the-art scores on OmniDocBench v1.6, Wild-O...

  42. ParVL: Parallel Scaling and Expandable Compute Allocation for Multimodal LLMs

    cs.CV 2026-08 conditional novelty 5.0 of 10

    ParVL scales MLLM computation by running multiple prefix-conditioned ViT and LLM branches over a shared backbone, improving average benchmark scores by 0.3 to 0.9 points and showing task-dependent vision-language allocation.

  43. SCLARO: A Dataset for Grounded Scenario-Level Scene Understanding and ScenarioCLIP for Benchmarking

    cs.CV 2025-11 conditional novelty 5.0 of 10

    An auto-generated 615K-image dataset with action, object, and relation annotations and a tri-level CLIP variant show in-distribution gains over PyramidCLIP, without external validation.

  44. Physical Plausibility Reasoning via HCM-GRPO: Empowering Compact Model for Superior Performance

    cs.CV 2025-11 conditional novelty 5.0 of 10

    A 2B multimodal model trained with HCM-GRPO, a GRPO variant with partial-credit rewards and hard-case oversampling, outperforms larger models on the authors' private physical-plausibility test set.

  45. Follow-Your-Instruction: A Comprehensive MLLM Agent for World Data Synthesis

    cs.CV 2025-08 conditional novelty 5.0 of 10

    An MLLM-driven pipeline that composes 3D scenes from assets, optimizes them with multi-view VLM feedback, and renders videos, yielding synthetic data that modestly improves several 2D, 3D, and 4D generative baselines.

  46. MMGraphRAG: Bridging Vision and Language with Interpretable Multimodal Knowledge Graphs

    cs.AI 2025-07 conditional novelty 5.0 of 10

    MMGraphRAG links scene-graph entities from images to text knowledge graph entities via SpecLink, and reports accuracy gains over naive RAG and GraphRAG on multimodal document QA.

  47. Ovis-U1 Technical Report

    cs.CV 2025-06 conditional novelty 5.0 of 10

    A 3B unified multimodal model with a diffusion decoder and bidirectional refiner achieves competitive understanding, generation, and editing benchmark scores.

  48. Med-Art: Diffusion Transformer for 2D Medical Text-to-Image Generation

    cs.CV 2025-06 conditional novelty 5.0 of 10

    Med-Art combines VLM-generated captions, LoRA fine-tuning of PixArt-alpha, and a pixel-space color-statistics loss to generate realistic 2D medical images from limited data.

  49. Taming the Untamed: Graph-Based Knowledge Retrieval and Reasoning for MLLMs to Conquer the Unknown

    cs.AI 2025-06 conditional novelty 5.0 of 10

    A multimodal knowledge graph benchmark for Monster Hunter: World, plus a training-free multi-agent graph retriever, improves MLLM accuracy on rare-domain questions from about 0.31 to 0.51 for GPT-4o.

  50. Towards an Explainable Comparison and Alignment of Feature Embeddings

    cs.LG 2025-06 conditional novelty 5.0 of 10

    SPEC finds and aligns the sample clusters that two embedding models capture differently by analyzing the eigenvectors of the difference of their kernel matrices.

  51. Multimodal Tabular Reasoning with Privileged Structured Information

    cs.LG 2025-06 conditional novelty 5.0 of 10

    An 8B multimodal LLM trained on 9k reasoning traces distilled from structured tables reaches state-of-the-art open-source accuracy on table-image question answering and fact verification.

  52. Instruction-augmented Multimodal Alignment for Image-Text and Element Matching

    cs.CV 2025-04 conditional novelty 5.0 of 10

    A fine-tuned multimodal score model with soft Q-Align scoring, element-conditioned prompts, and self-training on validation pseudo-labels takes first place in NTIRE 2025 Track 1 image-text alignment.

  53. COEF-VQ: Cost-Efficient Video Quality Understanding through a Cascaded Multimodal LLM Framework

    cs.CV 2024-12 conditional novelty 5.0 of 10

    A cascaded system that routes only high-entropy videos to a multimodal LLM achieves near-full-accuracy content moderation at a fraction of the GPU cost, and reduced inappropriate video views by 9.9% in an online A/B test.

  54. AnySynth: Harnessing the Power of Image Synthetic Data Generation for Generalized Vision-Language Tasks

    cs.CV 2024-11 conditional novelty 5.0 of 10

    AnySynth is a single synthetic-data pipeline that produces layouts, images, and annotations for multiple vision tasks, and its data improves benchmark scores by small but consistent margins.

  55. Boogu-Image-0.1: Boosting Open Agentic Multimodal Generation via Understanding under a Minimal Budget

    cs.CV 2026-07 conditional novelty 4.0 of 10

    An open-source image-generation family shows that agentic prompt rewriting and a stronger text encoder can lift quality to near closed-source levels with only 208.62M images and about $400K of training compute.

  56. Describe Anything Model for Visual Question Answering on Text-rich Images

    cs.CV 2025-07 conditional novelty 4.0 of 10

    DAM-QA aggregates answers from full-image and sliding-window views of the Describe Anything Model with a weighted vote, improving text-rich VQA on some benchmarks but not all.

  57. SAILViT: Towards Robust and Generalizable Visual Backbones for MLLMs via Gradual Feature Refinement

    cs.CV 2025-07 conditional novelty 4.0 of 10

    A three-stage coarse-to-fine training recipe for vision backbones produces consistent benchmark gains for lightweight multimodal LLMs.

  58. NTIRE 2025 challenge on Text to Image Generation Model Quality Assessment

    cs.CV 2025-05 conditional novelty 4.0 of 10

    The NTIRE 2025 challenge report compares 20 methods for fine-grained text-to-image quality assessment, introduces the EvalMuse-Structure dataset, and finds every participating team outperformed the baselines.

  59. Valley2: Exploring Multimodal Models with Scalable Vision-Language Design

    cs.CV 2025-01 conditional novelty 4.0 of 10

    Valley2, a 7B-scale open-source multimodal model, reports second-best OpenCompass average (67.4) among sub-10B models and the highest score (79.66) on its own in-house Ecom-VQA benchmark.

  60. POINTS1.5: Building a Vision-Language Model towards Real World Applications

    cs.CV 2024-12 conditional novelty 4.0 of 10

    POINTS1.5-7B, a vision-language model with a NaViT-style encoder, bilingual data, and filtered instruction tuning, ranks first on OpenCompass among sub-10B models.

Pith tools