Pith. sign in

REVIEW 1 major objections 1 minor 82 cited by

InternLM-XComposer2: Mastering Free-form Text-Image Composition and Comprehension in Vision-Language Large Model

T0 review · 1 major / 1 minor · reviewed 2026-05-17 · grok-4.3

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read InternLM-XComposer2 generates custom interleaved text-image content by applying LoRA parameters only to image tokens.

desk verdict InternLM-XComposer2 adds Partial LoRA only to image tokens to extend InternLM2-7B for text-image tasks, but the preservation of language knowledge lacks supporting ablations. read the letter →

arxiv 2401.16420 v1 pith:YAOHT3YA submitted 2024-01-29 cs.CV cs.CL

Xiaoyi Dong , Pan Zhang , Yuhang Zang , Yuhang Cao , Bin Wang , Linke Ouyang , Xilin Wei , Songyang Zhang
show 15 more authors
Haodong Duan Maosong Cao Wenwei Zhang Yining Li Hang Yan Yang Gao Xinyue Zhang Wei Li Jingwen Li Kai Chen Conghui He Xingcheng Zhang Yu Qiao Dahua Lin Jiaqi Wang
This is my paper · ORCID
classification cs.CVcs.CL
keywords vision-languagemodeltext-imagecompositionpartialLoRAmultimodalgenerationfree-formcontentcreationInternLM-XComposer2interleavedtextandimages
checked against Cost.FunctionalEquationFoundation.DimensionForcing
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper presents InternLM-XComposer2 as a vision-language model that creates free-form text-image compositions from inputs such as outlines, detailed text, or reference images. It introduces a Partial LoRA method that adds adaptation parameters solely to image tokens while leaving the pre-trained language model untouched. This design aims to deliver both accurate vision understanding and fluent, high-quality text composition in long multimodal outputs. The model, based on a 7B InternLM2 backbone, is shown to exceed prior multimodal systems on benchmarks and to match or exceed GPT-4V and Gemini Pro on selected tasks. The central argument is that selective adaptation of vision components enables strong multimodal generation without sacrificing linguistic capability.

What carries the argument

Partial LoRA (PLoRA) that applies LoRA parameters exclusively to image tokens to balance precise vision understanding with literary-quality text composition.

What would settle it

A measurable drop in performance on pure language-only benchmarks after PLoRA training would show that language knowledge was not preserved.

Watch

Extended reading notes

Core claim

InternLM-XComposer2 demonstrates that applying additional LoRA parameters exclusively to image tokens produces a model capable of high-quality free-form text-image composition and comprehension, outperforming existing multimodal models and matching or surpassing GPT-4V and Gemini Pro on certain benchmarks while preserving the integrity of the pre-trained language knowledge.

Load-bearing premise

Adding LoRA parameters only to image tokens preserves the original language model's knowledge while still enabling strong vision understanding and text-image generation.

Editorial extensions

If this is right

  • The model can produce long, interleaved multimodal documents from outlines or reference images.
  • Vision-language understanding reaches or exceeds GPT-4V and Gemini Pro levels on selected evaluations.
  • High-quality content creation becomes possible without full fine-tuning of the language backbone.
  • The same PLoRA pattern may extend to other base language models of similar size.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Selective tuning of vision components could reduce the risk of language degradation seen in full multimodal fine-tuning.
  • This separation of adaptation might allow smaller teams to build capable multimodal systems on top of existing open language models.
  • Testing PLoRA on tasks that require very long context or creative writing would clarify how far the preserved language skill extends.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

1 major / 1 minor

Summary. The paper presents InternLM-XComposer2, a 7B-parameter vision-language model built on InternLM2 that introduces Partial LoRA (PLoRA) to apply additional LoRA parameters exclusively to image tokens. This design is claimed to preserve the base model's pre-trained language knowledge while enabling high-quality free-form interleaved text-image generation and comprehension from inputs such as outlines, textual specifications, and reference images. The manuscript reports that the model significantly outperforms prior multimodal systems and matches or exceeds GPT-4V and Gemini Pro on selected vision-language benchmarks, with the model weights publicly released.

Significance. If the central performance claims and the PLoRA preservation hypothesis are substantiated, the work would be significant for providing a lightweight, modular route to extend strong language models into multimodal composition tasks without full fine-tuning. The public release of the 7B model series would further enable reproducible research on controllable text-image generation.

major comments (1)
  1. [§3.2] §3.2 and abstract: The central design claim that PLoRA (LoRA applied only to image tokens) preserves InternLM2-7B's pre-trained language knowledge while adding vision capabilities is asserted without supporting ablation evidence. No results are shown for language-only benchmarks (e.g., MMLU, GSM8K) before versus after PLoRA, nor any direct comparison of PLoRA versus standard LoRA applied to all tokens. Because cross-attention layers still mix modalities, this isolation assumption is not guaranteed by architecture alone and is load-bearing for the claimed balance between vision understanding and literary text composition.
minor comments (1)
  1. [Abstract] Abstract and experimental section: The superiority claims reference various benchmarks but provide no details on data splits, evaluation protocols, statistical significance, or exact metric definitions, making it difficult to assess the strength of the reported gains over baselines.

Simulated Author's Rebuttal

1 responses · 0 unresolved

We thank the referee for the constructive feedback. We address the major comment below and describe the changes planned for the revised manuscript.

read point-by-point responses

  1. Referee: [§3.2] §3.2 and abstract: The central design claim that PLoRA (LoRA applied only to image tokens) preserves InternLM2-7B's pre-trained language knowledge while adding vision capabilities is asserted without supporting ablation evidence. No results are shown for language-only benchmarks (e.g., MMLU, GSM8K) before versus after PLoRA, nor any direct comparison of PLoRA versus standard LoRA applied to all tokens. Because cross-attention layers still mix modalities, this isolation assumption is not guaranteed by architecture alone and is load-bearing for the claimed balance between vision understanding and literary text composition.

    Authors: We appreciate the referee highlighting the need for stronger empirical support. The PLoRA design applies LoRA updates exclusively to image tokens while keeping base InternLM2 weights frozen for text tokens, which is intended to limit interference with pre-trained language abilities. We agree that direct ablations would strengthen the manuscript. In the revision we will add language-only benchmark results (MMLU, GSM8K) comparing the original InternLM2-7B to the PLoRA-adapted model to quantify preservation. We will also include a side-by-side comparison of PLoRA versus standard LoRA applied to all tokens, showing advantages for text composition quality. Regarding modality mixing through attention layers, although cross-modal interactions exist, the position-specific LoRA application ensures that core language parameters and the modeling head for pure text sequences remain unchanged, which is consistent with the observed high-quality long-text generation performance. revision: yes

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity; empirical claims rest on external benchmarks

full rationale

The paper introduces InternLM-XComposer2 with a Partial LoRA (PLoRA) mechanism applied selectively to image tokens on top of InternLM2-7B. Central claims of superior free-form text-image composition and comprehension are supported by reported experimental results on various benchmarks, including direct comparisons to GPT-4V and Gemini Pro. No equations, self-definitional loops, fitted inputs renamed as predictions, or load-bearing self-citations that reduce the reported outcomes to the paper's own inputs by construction appear in the provided text. The approach is presented as an architectural proposal with empirical validation against independent external references.

Assumptions & free parameters 1 free parameters · 0 assumptions · 0 invented entities

The central claim depends on the empirical effectiveness of Partial LoRA for modality balance, an approach chosen without detailed theoretical derivation in the provided abstract.

free parameters (1)
  • Partial LoRA rank and scaling
    Hyperparameters controlling the added adaptation matrices applied only to image tokens; values chosen to preserve language capabilities.

how reviews work

0 comments
Cite this review

Pith. "Pith review of InternLM-XComposer2: Mastering Free-form Text-Image Composition and Comprehension in Vision-Language Large Model." pith.science (2026). https://pith.science/paper/YAOHT3YA

@misc{pith2026240116420,
  author       = {Pith},
  title        = {Pith review of: InternLM-XComposer2: Mastering Free-form Text-Image Composition and Comprehension in Vision-Language Large Model},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/YAOHT3YA}},
  note         = {Machine review of arXiv:2401.16420}
}
read the original abstract

We introduce InternLM-XComposer2, a cutting-edge vision-language model excelling in free-form text-image composition and comprehension. This model goes beyond conventional vision-language understanding, adeptly crafting interleaved text-image content from diverse inputs like outlines, detailed textual specifications, and reference images, enabling highly customizable content creation. InternLM-XComposer2 proposes a Partial LoRA (PLoRA) approach that applies additional LoRA parameters exclusively to image tokens to preserve the integrity of pre-trained language knowledge, striking a balance between precise vision understanding and text composition with literary talent. Experimental results demonstrate the superiority of InternLM-XComposer2 based on InternLM2-7B in producing high-quality long-text multi-modal content and its exceptional vision-language understanding performance across various benchmarks, where it not only significantly outperforms existing multimodal models but also matches or even surpasses GPT-4V and Gemini Pro in certain assessments. This highlights its remarkable proficiency in the realm of multimodal understanding. The InternLM-XComposer2 model series with 7B parameters are publicly available at https://github.com/InternLM/InternLM-XComposer.

Discussion (0). Continue with ORCID to comment.

Forward citations

Showing 60 of 82 Pith papers that cite this

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. See all 82 Pith citations

  1. MentalThink: Shaping Thoughts in Mental SVG World

    cs.AI 2026-07 conditional novelty 7.0 of 10

    MLLMs that generate and render SVG sketches as multi-turn intermediate reasoning steps reach 55.1% on VSIBench and 76.0% on MindCube, far above the Qwen2.5-VL-7B backbone.

  2. One Generator, Any Process: LLM-Conditioning for the LHC

    hep-ph 2026-06 unverdicted novelty 7.0 of 10

    LLM embeddings condition generative networks for LHC events, yielding faster convergence, higher quality, and generalization to unseen processes.

  3. Beyond the Current Observation: Evaluating Multimodal Large Language Models in Controllable Non-Markov Games

    cs.CV 2026-06 unverdicted novelty 7.0 of 10

    RNG-Bench evaluates MLLMs on hidden-observation reconstruction in non-Markov games, finds forgetting as the dominant error source, and shows fine-tuning on optimal rollouts improves performance with transfer to other ...

  4. CaMo: Camera Motion Grounded Evaluation and Training for Vision-Language Models

    cs.CV 2026-05 unverdicted novelty 7.0 of 10

    Proposes Spatial Narrative Score (SNS) evaluation for VLMs' camera motion understanding and introduces CaMo model achieving consistent performance on SNS and direct QA.

  5. GPT4Scene: Understand 3D Scenes from Videos with Vision-Language Models

    cs.CV 2025-01 conditional novelty 7.0 of 10

    Consistent object IDs across video frames and a reconstructed bird's-eye view let large VLMs, and a fine-tuned 7B model, reach state-of-the-art results on ScanNet 3D question answering, captioning, and grounding.

  6. Polymath: A Challenging Multi-modal Mathematical Reasoning Benchmark

    cs.AI 2024-10 unverdicted novelty 7.0 of 10

    PolyMATH is a new 5,000-image benchmark where top MLLMs reach at most 41 percent accuracy on multi-modal mathematical reasoning, with ablation showing minimal gain from text over images.

  7. Graph it first! Enabling Reasoning on Long-form Egocentric Videos through Scene Graphs

    cs.CV 2026-06 unverdicted novelty 6.0 of 10

    Introduces temporally grounded EgoSGs to convert long egocentric videos into compact symbolic text for MLLM-based VQA, claiming SOTA results on HD-EPIC without subsampling.

  8. Qwen-RobotWorld Technical Report: Unifying Embodied World Modeling through Language-Conditioned Video Generation

    cs.CV 2026-06 unverdicted novelty 6.0 of 10

    Qwen-RobotWorld is a language-conditioned video world model using Double-Stream MMDiT, an 8.6M-frame embodied corpus, and progressive curriculum training that ranks first on EWMBench and DreamGen Bench.

  9. E-VLA: Event-Augmented Vision-Language-Action Model for Dark and Blurred Scenes

    cs.CV 2026-04 conditional novelty 6.0 of 10

    E-VLA integrates event streams directly into VLA models via lightweight fusion, raising Pick-Place success from 0% to 60-90% at 20 lux and from 0% to 20-25% under severe motion blur.

  10. Predictive Modeling for High Impact Active Learning Classrooms

    physics.ed-ph 2026-03 unverdicted novelty 6.0 of 10

    Four classroom-time variables predict physics concept learning, and classes with 10–20% group worksheets, 20–40% group clickers, and ≥2 student questions per hour show effect sizes above 2.

  11. ReMoT: Reinforcement Learning with Motion Contrast Triplets

    cs.CV 2026-02 conditional novelty 6.0 of 10

    Training a 4B vision-language model on rule-generated motion-contrast triplets with GRPO lifts spatio-temporal QA accuracy by about 17 points on the authors' own benchmark and by smaller margins on standard benchmarks.

  12. SHALE: A Scalable Benchmark for Fine-grained Hallucination Evaluation in LVLMs

    cs.CV 2025-08 conditional novelty 6.0 of 10

    SHALE is a synthetic, scalable benchmark that evaluates faithful and factual hallucination in large vision-language models across 12 visual and 6 factual categories, in clean and perturbed settings.

  13. Bridging the Gap in Vision Language Models in Identifying Unsafe Concepts Across Modalities

    cs.CR 2025-07 conditional novelty 6.0 of 10

    Vision-language models consistently recognize unsafe content better from text than from images, and a simplified reinforcement learning fine-tune narrows that gap.

  14. A Visual Leap in CLIP Compositionality Reasoning through Generation of Counterfactual Sets

    cs.CV 2025-07 conditional novelty 6.0 of 10

    Block-based diffusion generation of counterfactual image-text sets, combined with a set-aware loss, improves CLIP's compositional reasoning over several benchmarks, but the paper overstates one benchmark result and sh...

  15. NOVO: Unlearning-Compliant Vision Transformers

    cs.CV 2025-07 conditional novelty 6.0 of 10

    NOVO is a vision transformer that forgets classes at inference time by removing learned class keys, trained with simulated unlearning to generalize to any forget set.

  16. LLaVA-Scissor: Token Compression with Semantic Connected Components for Video LLMs

    cs.CV 2025-06 conditional novelty 6.0 of 10

    A training-free token compression method using semantic connected components in space and time keeps video understanding accuracy high even when retaining only 5-10% of visual tokens.

  17. CFBenchmark-MM: Chinese Financial Assistant Benchmark for Multimodal Large Language Model

    cs.CL 2025-06 conditional novelty 6.0 of 10

    A 9,356-pair Chinese multimodal financial benchmark reveals that state-of-the-art multimodal LLMs, including GPT-4V, still score below 53% on objective and 39% on subjective financial chart tasks.

  18. Rethinking Multilingual Vision-Language Translation: Dataset, Evaluation, and Adaptation

    cs.CV 2025-06 conditional novelty 6.0 of 10

    The authors introduce AibTrans, a multilingual image-text translation benchmark, show that common translation metrics mislead on dense images, and find that balanced multilingual fine-tuning preserves generalization b...

  19. Controllable Coupled Image Generation via Diffusion Models

    cs.CV 2025-06 reject novelty 6.0 of 10

    A cross-attention control method that couples backgrounds across multiple generated images by blending LLM-extracted background and entity prompts with a time-varying weight optimized for background similarity and tex...

  20. ReFoCUS: Reinforcement-guided Frame Optimization for Contextual Understanding

    cs.CV 2025-06 conditional novelty 6.0 of 10

    A reinforcement-learned frame selection policy, trained with reward margins from a reference video-LLM, improves video QA accuracy of LLaVA-OV and InternVL3 across several benchmarks.

  21. Spoken question answering for visual queries

    eess.AS 2025-05 conditional novelty 6.0 of 10

    A LLaVA-style model with an added Whisper speech encoder answers spoken questions about images, trained on TTS-synthesized speech and reaching near the text-input baseline.

  22. AutoJudger: An Agent-Driven Framework for Efficient Benchmarking of MLLMs

    cs.CL 2025-05 conditional novelty 6.0 of 10

    An agent-driven framework adaptively selects a small subset of benchmark questions for MLLMs, preserving over 90% ranking accuracy with roughly 4-5% of the data.

  23. MLLM-Guided VLM Fine-Tuning with Joint Inference for Zero-Shot Composed Image Retrieval

    cs.CV 2025-05 conditional novelty 6.0 of 10

    MVFT-JI trains a Q-Former VLM with two MLLM-generated retrieval tasks and fuses VLM and MLLM similarities at inference, achieving state-of-the-art zero-shot composed image retrieval on three benchmarks.

  24. MathFlow: Enhancing the Perceptual Flow of MLLMs for Visual Mathematical Problems

    cs.CV 2025-03 unverdicted novelty 6.0 of 10

    MathFlow decouples perception and inference stages in MLLMs for visual math, with a dedicated perception model delivering gains on the FlowVerse benchmark when paired with existing reasoners.

  25. MME-Industry: A Cross-Industry Multimodal Evaluation Benchmark

    cs.CL 2025-01 conditional novelty 6.0 of 10

    MME-Industry is a new 21-domain, bilingual multiple-choice benchmark for measuring multimodal LLMs in industrial settings, with top models scoring around 75 to 79 percent.

  26. Analyzing and Boosting the Power of Fine-Grained Visual Recognition for Multi-modal Large Language Models

    cs.CV 2025-01 reject novelty 6.0 of 10

    Finedefics improves fine-grained image classification in multimodal LLMs by contrastively aligning image, attribute, and category embeddings, though its headline gains are measured against zero-shot baselines rather t...

  27. Global Semantic-Guided Sub-image Feature Weight Allocation in High-Resolution Large Vision-Language Models

    cs.CV 2025-01 conditional novelty 6.0 of 10

    A semantic-similarity weighting module for sub-images improves high-resolution vision-language model performance, pending clarification of whether evaluation benchmarks overlap with training data.

  28. GeoPixel: Pixel Grounding Large Multimodal Model in Remote Sensing

    cs.CV 2025-01 conditional novelty 6.0 of 10

    GeoPixel brings pixel-level grounding to remote sensing large multimodal models for the first time, with a new dataset and benchmark built from iSAID.

  29. T2ISafety: Benchmark for Assessing Fairness, Toxicity, and Privacy in Image Generation

    cs.CL 2025-01 conditional novelty 6.0 of 10

    T2ISafety is a large annotated benchmark plus a fine-tuned MLLM evaluator (ImageGuard) for measuring toxicity, privacy, and fairness in text-to-image models.

  30. Facial Dynamics in Video: Instruction Tuning for Improved Facial Expression Perception and Contextual Awareness

    cs.CV 2025-01 conditional novelty 6.0 of 10

    The paper presents FDA, a manually annotated dataset, FaceTrack-MM, a face-tracking video MLLM, FEC-Bench, a benchmark, and TEM, a ChatGPT-based metric, all for dynamic facial expression captioning.

  31. LongViTU: Instruction Tuning for Long-Form Video Understanding

    cs.CV 2025-01 conditional novelty 6.0 of 10

    LongViTU, an automatically generated 121k-pair video QA dataset with 4.6-minute average certificate length, provides modest SFT gains on long-video benchmarks.

  32. JELLY: Joint Emotion Recognition and Context Reasoning with LLMs for Conversational Speech Synthesis

    cs.CL 2025-01 conditional novelty 6.0 of 10

    JELLY fine-tunes an LLM with partial LoRA adapters and an emotion-aware Q-former to predict and synthesize emotionally appropriate conversational speech from speech alone.

  33. EAGLE: Enhanced Visual Grounding Minimizes Hallucinations in Instructional Multimodal Models

    cs.CV 2025-01 conditional novelty 6.0 of 10

    Tuning a CLIP-style visual encoder with masked-pooling contrastive losses on instance segmentation data reduces hallucinations in downstream instruction-tuned vision-language models when the encoder is swapped in with...

  34. IllusionBench+: A Large-scale and Comprehensive Benchmark for Visual Illusion Understanding in Vision-Language Models

    cs.CV 2025-01 conditional novelty 6.0 of 10

    IllusionBench+ is a 1,051-image benchmark with 5,548 QA pairs and golden descriptions for testing vision-language models on classic, real-scene, Ishihara, and trap visual illusions, and it finds top models such as GPT...

  35. MM-MoralBench: A MultiModal Moral Evaluation Benchmark for Large Vision-Language Models

    cs.CV 2024-12 unverdicted novelty 6.0 of 10

    MM-MoralBench is a new multimodal benchmark that evaluates over 20 LVLMs on moral judgment, classification, and response tasks, finding pronounced divergence from human consensus and limited benefits from scaling or c...

  36. ST$^3$: Accelerating Multimodal Large Language Model by Spatial-Temporal Visual Token Trimming

    cs.CV 2024-12 conditional novelty 6.0 of 10

    ST3 progressively removes inattentive visual tokens across network layers and across generation steps, achieving about 2x faster MLLM inference with roughly 30 to 50 percent of the original KV cache memory.

  37. ICM-Assistant: Instruction-tuning Multimodal Large Language Models for Rule-based Explainable Image Content Moderation

    cs.CV 2024-12 conditional novelty 6.0 of 10

    A rule-decomposition data pipeline and 246K question-answer pairs let instruction-tuned multimodal LLMs classify and explain image content moderation more accurately than their untuned versions.

  38. A High-Quality Text-Rich Image Instruction Tuning Dataset via Hybrid Instruction Generation

    cs.CV 2024-12 conditional novelty 6.0 of 10

    A hybrid human/GPT-4o pipeline plus two new filtering scores produces a 424k-pair text-rich image instruction dataset that improves fine-tuned multimodal models on OCR and document understanding benchmarks.

  39. AI-generated Image Quality Assessment in Visual Communication

    cs.CV 2024-12 conditional novelty 6.0 of 10

    AIGI-VC is a 2,500-image benchmark for judging AI-generated ads on clarity and emotional impact, and current IQA metrics and open LMMs mostly fail at it.

  40. EarthDial: Turning Multi-sensory Earth Observations to Interactive Dialogues

    cs.CV 2024-12 conditional novelty 6.0 of 10

    EarthDial is a 4B-parameter remote sensing chatbot trained on 11.11M instruction pairs to handle multi-resolution, multi-spectral, and multi-temporal satellite imagery, and it reports gains over prior VLMs on dozens o...

  41. Consistency of Compositional Generalization across Multiple Levels

    cs.CV 2024-12 conditional novelty 6.0 of 10

    Large vision-language models score below 50 percent on a new cross-level compositional consistency benchmark, and a progressive meta-optimization training framework improves consistency on several VQA and TVG baselines.

  42. V2PE: Improving Multimodal Long-Context Capability of Vision-Language Models with Variable Visual Position Encoding

    cs.CV 2024-12 conditional novelty 6.0 of 10

    V2PE assigns visual tokens smaller and variable positional increments than text tokens, which allows a 2B vision-language model to effectively process multimodal sequences up to 1M tokens.

  43. Rethinking Comprehensive Benchmark for Chart Understanding: A Perspective from Scientific Literature

    cs.CL 2024-12 conditional novelty 6.0 of 10

    A new benchmark built from real scientific paper charts, including flowcharts and context-dependent questions, shows large multimodal models perform far below human level on chart understanding.

  44. Chimera: Improving Generalist Model with Domain-Specific Experts

    cs.CV 2024-12 conditional novelty 6.0 of 10

    Chimera fuses frozen domain-expert encoders into a generalist multimodal LLM via routing and a 30% masking of general tokens, lifting InternVL2-8B from 61.6 to 64.9 on MathVista and from 31.3 to 32.4 on MathVerse.

  45. Align-KD: Distilling Cross-Modal Alignment Knowledge for Mobile Vision-Language Model Enhancement

    cs.CV 2024-12 conditional novelty 6.0 of 10

    Align-KD distills the teacher's first-layer text-to-image attention and focused vision embeddings into a 1.7B student VLM, improving average benchmark scores by up to 2.0 points.

  46. OBI-Bench: Can LMMs Aid in Study of Ancient Script on Oracle Bones?

    cs.CV 2024-12 conditional novelty 6.0 of 10

    OBI-Bench evaluates 23 large multimodal models on five oracle bone inscription tasks and finds they lag on fine-grained perception but approach untrained-human level in deciphering.

  47. Insight-V: Exploring Long-Chain Visual Reasoning with Multimodal Large Language Models

    cs.CV 2024-11 conditional novelty 6.0 of 10

    Insight-V uses machine-generated long-chain reasoning data, a reasoning-plus-summary multi-agent setup, and iterative DPO to improve visual reasoning scores of multimodal LLMs.

  48. RAR: Retrieving And Ranking Augmented MLLMs for Visual Recognition

    cs.CV 2024-03 unverdicted novelty 6.0 of 10

    RAR combines CLIP retrieval with MLLM ranking to improve few-shot and zero-shot fine-grained visual recognition on 5 benchmarks, 11 few-shot datasets, and 2 detection tasks.

  49. Qwen-Audio-VAE Technical Report

    eess.AS 2026-07 conditional novelty 5.0 of 10

    A 12.5 Hz continuous audio VAE reconstructs speech, music, and sound well while encoding 64×30s clips in 541 ms after latency-aware encoder pruning.

  50. ERA: Entropy-Guided Visual Token Pruning with Rectified Attention for Efficient MLLMs

    cs.CV 2026-06 unverdicted novelty 5.0 of 10

    ERA proposes entropy-guided token pruning with bias-aware recycling and logit rectification to compress visual inputs in MLLMs while mitigating attention collapse.

  51. NoisyGRPO: Incentivizing Multimodal CoT Reasoning via Noise Injection and Bayesian Estimation

    cs.CV 2025-10 unverdicted novelty 5.0 of 10

    NoisyGRPO is an RL framework that perturbs visual inputs with Gaussian noise for exploration and computes trajectory advantages via Bayesian posterior fusion of noise prior and reward likelihood to improve multimodal ...

  52. DinoCompanion: An Attachment-Theory Informed Multimodal Robot for Emotionally Responsive Child-AI Interaction

    cs.AI 2025-06 reject novelty 5.0 of 10

    A child-facing robot trained with an attachment-theory-informed preference optimization is claimed to beat GPT-4o and Gemini-2.5-Pro on a new ten-competency benchmark, though evaluation and derivation issues undermine...

  53. ChartReasoner: Code-Driven Modality Bridging for Long-Chain Reasoning in Chart Question Answering

    cs.CL 2025-06 conditional novelty 5.0 of 10

    ChartReasoner converts charts into executable ECharts code, distills long-chain reasoning traces from that code, and trains a 7B multimodal model with SFT and GRPO to improve chart question answering.

  54. Does Your 3D Encoder Really Work? When Pretrain-SFT from 2D VLMs Meets 3D VLMs

    cs.CV 2025-06 conditional novelty 5.0 of 10

    3D scene-centric VLMs underutilize the 3D encoder, overfit to linguistic and answer-frequency shortcuts, and the proposed 3D-RDQA dataset helps expose and mitigate this problem.

  55. Seeing Far and Clearly: Mitigating Hallucinations in MLLMs with Attention Causal Decoding

    cs.CV 2025-05 conditional novelty 5.0 of 10

    FarSight adds upper-triangular negative biases to the causal mask to absorb outlier-token attention, reducing hallucinations in MLLMs without training.

  56. Learning Conformal Abstention Policies for Adaptive Risk Management in Large Language and Vision-Language Models

    cs.LG 2025-02 reject novelty 5.0 of 10

    CAP tunes conformal thresholds with RL to switch between single answers, sets, and abstention, but its test-set-fitting undermines the claimed statistical guarantees.

  57. RDMM: Fine-Tuned LLM Models for On-Device Robotic Decision Making with Enhanced Contextual Awareness in Specific Domains

    cs.RO 2025-01 conditional novelty 5.0 of 10

    Fine-tuned, quantized LLMs achieve about 93% accuracy on an internal RoboCup@Home planning benchmark and run locally on Jetson hardware.

  58. Exploring the Role of Explicit Temporal Modeling in Multimodal Large Language Models for Video Understanding

    cs.CV 2025-01 conditional novelty 5.0 of 10

    Adding a trainable convolutional temporal encoder to LLaVA-style video models improves benchmark scores and allows heavy frame compression, but the controlled evidence for the causal claim is weak.

  59. T-Stars-Poster: A Framework for Product-Centric Advertising Image Design

    cs.CV 2025-01 conditional novelty 5.0 of 10

    A four-stage product-centric framework automatically creates advertising posters from a product photo, taglines, and target size, with better visual-quality scores and about 3 percent higher click-through rates in Tao...

  60. LLaVA-Octopus: Unlocking Instruction-Driven Adaptive Projector Fusion for Video Understanding

    cs.CV 2025-01 unverdicted novelty 5.0 of 10

    LLaVA-Octopus introduces instruction-driven adaptive fusion of multiple visual projectors in a multimodal LLM to improve video understanding performance.

See all 82 Pith citations

Reference graph

Works this paper leans on

105 extracted references · 105 canonical work pages · cited by 82 Pith papers (see all)

  1. [1]

    Nocaps: Novel object captioning at scale

    Harsh Agrawal, Karan Desai, Yufei Wang, Xinlei Chen, Rishabh Jain, Mark Johnson, Dhruv Batra, Devi Parikh, Stefan Lee, and Peter Anderson. Nocaps: Novel object captioning at scale. In Proceedings of the IEEE/CVF inter- national conference on computer vision, pages 8948–8957,

    work page

  2. [2]

    Flamingo: a visual language model for few-shot learning,

    Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, An- toine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katie Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob Menick, Se- bastian Borgeaud, Andrew Brock, Aida Nematzadeh, Sa- hand Sharifzadeh, Mikolaj Bink...

    work page

  3. [3]

    MathQA: Towards Interpretable Math Word Problem Solving with Operation-Based Formalisms

    Aida Amini, Saadia Gabriel, Peter Lin, Rik Koncel- Kedziorski, Yejin Choi, and Hannaneh Hajishirzi. Mathqa: Towards interpretable math word problem solv- ing with operation-based formalisms. arXiv preprint arXiv:1905.13319, 2019. 4

    work page Pith review arXiv 1905

  4. [4]

    Lawrence Zitnick, and Devi Parikh

    Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C. Lawrence Zitnick, and Devi Parikh. Vqa: Visual question answering. In International Conference on Computer Vision (ICCV), 2015. 4

    work page 2015

  5. [5]

    Openflamingo: An open- source framework for training large autoregressive vision- language models

    Anas Awadalla, Irena Gao, Josh Gardner, Jack Hessel, Yusuf Hanafy, Wanrong Zhu, Kalyani Marathe, Yonatan Bitton, Samir Gadre, Shiori Sagawa, Jenia Jitsev, Simon Kornblith, Pang Wei Koh, Gabriel Ilharco, Mitchell Worts- man, and Ludwig Schmidt. Openflamingo: An open- source framework for training large autoregressive vision- language models. arXiv.org, 2023. 3

    work page 2023

  6. [6]

    Qwen-vl: A frontier large vision-language model with versatile abilities

    Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A frontier large vision-language model with versatile abilities. arXiv.org, 2023. 2, 3

    work page 2023

  7. [7]

    Baichuan 2: Open large-scale language models

    Baichuan. Baichuan 2: Open large-scale language models. arXiv.org, 2023. 2, 3

    work page 2023

  8. [8]

    Language models are few-shot learners

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Sub- biah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakan- tan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in Neu- ral Information Processing Systems (NeurIPS) , 33:1877– 1901, 2020. 2

    work page 1901

Show all 105 references
  1. [9]

    Minigpt-v2: large language model as a unified interface for vision-language multi-task learning

    Jun Chen, Deyao Zhu, Xiaoqian Shen, Xiang Li, Zechu Liu, Pengchuan Zhang, Raghuraman Krishnamoorthi, Vikas Chandra, Yunyang Xiong, and Mohamed Elhoseiny. Minigpt-v2: large language model as a unified interface for vision-language multi-task learning. arXiv preprint arXiv:2310....

    2023 arXiv

  2. [10]

    Shikra: Unleashing multimodal llm’s referential dialogue magic

    Keqin Chen, Zhao Zhang, Weili Zeng, Richong Zhang, Feng Zhu, and Rui Zhao. Shikra: Unleashing multimodal llm’s referential dialogue magic. arXiv.org, 2023. 3

    2023

  3. [11]

    Sharegpt4v: Improving large multi-modal models with better captions

    Lin Chen, Jisong Li, Xiaoyi Dong, Pan Zhang, Conghui He, Jiaqi Wang, Feng Zhao, and Dahua Lin. Sharegpt4v: Improving large multi-modal models with better captions. arXiv preprint arXiv:2311.12793, 2023. 3, 4 7

    2023 arXiv

  4. [12]

    Pali-x: On scaling up a multilingual vision and language model, 2023

    Xi Chen, Josip Djolonga, Piotr Padlewski, Basil Mustafa, Soravit Changpinyo, Jialin Wu, Carlos Riquelme Ruiz, Sebastian Goodman, Xiao Wang, Yi Tay, Siamak Shak- eri, Mostafa Dehghani, Daniel Salz, Mario Lucic, Michael Tschannen, Arsha Nagrani, Hexiang Hu, Mandar Joshi, Bo Pang...

    2023

  5. [13]

    Lawrence Zitnick

    Xinlei Chen, Hao Fang, Tsung-Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollar, and C. Lawrence Zitnick. Microsoft coco captions: Data collection and eval- uation server, 2015. 4

    2015

  6. [14]

    Pali-3 vision language models: Smaller, faster, stronger, 2023

    Xi Chen, Xiao Wang, Lucas Beyer, Alexander Kolesnikov, Jialin Wu, Paul V oigtlaender, Basil Mustafa, Sebas- tian Goodman, Ibrahim Alabdulmohsin, Piotr Padlewski, Daniel Salz, Xi Xiong, Daniel Vlasic, Filip Pavetic, Keran Rong, Tianli Yu, Daniel Keysers, Xiaohua Zhai, and Radu ...

    2023

  7. [15]

    Pali: A jointly-scaled multilingual language- image model, 2023

    Xi Chen, Xiao Wang, Soravit Changpinyo, AJ Piergio- vanni, Piotr Padlewski, Daniel Salz, Sebastian Goodman, Adam Grycner, Basil Mustafa, Lucas Beyer, Alexander Kolesnikov, Joan Puigcerver, Nan Ding, Keran Rong, Has- san Akbari, Gaurav Mishra, Linting Xue, Ashish Thapliyal, Jam...

    2023

  8. [16]

    Gonzalez, Ion Stoica, and Eric P

    Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhang- hao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023. 2, 4

    2023

  9. [17]

    Palm: Scaling language modeling with pathways

    Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. arXiv.org, 2022. 2

    2022

  10. [18]

    Opencompass: A univer- sal evaluation platform for foundation models

    OpenCompass Contributors. Opencompass: A univer- sal evaluation platform for foundation models. https: / / github . com / open - compass / opencompass,

  11. [19]

    Qwen-vl-plus

    QWen Contributors. Qwen-vl-plus. https : / / huggingface . co / spaces / Qwen / Qwen - VL - Plus, year=2023. 2

    2023

  12. [20]

    Xtuner: A toolkit for efficiently fine-tuning llm

    XTuner Contributors. Xtuner: A toolkit for efficiently fine-tuning llm. https://github.com/InternLM/ xtuner, 2023. 3

    2023

  13. [21]

    Instructblip: Towards general- purpose vision-language models with instruction tuning,

    Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. Instructblip: Towards general- purpose vision-language models with instruction tuning,

  14. [22]

    Bert: Pre-training of deep bidirectional trans- formers for language understanding

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional trans- formers for language understanding. arXiv.org, 2018. 2

    2018

  15. [23]

    Dreamllm: Synergistic multimodal com- prehension and creation

    Runpei Dong, Chunrui Han, Yuang Peng, Zekun Qi, Zheng Ge, Jinrong Yang, Liang Zhao, Jianjian Sun, Hongyu Zhou, Haoran Wei, Xiangwen Kong, Xiangyu Zhang, Kaisheng Ma, and Li Yi. Dreamllm: Synergistic multimodal com- prehension and creation. arXiv preprint arXiv:2309.11499,

  16. [24]

    Danny Driess, Fei Xia, Mehdi S. M. Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, Wenlong Huang, Yevgen Chebotar, Pierre Sermanet, Daniel Duck- worth, Sergey Levine, Vincent Vanhoucke, Karol Hausman, Marc Toussai...

    2023 arXiv

  17. [25]

    Glm: General language model pretraining with autoregressive blank infilling

    Zhengxiao Du, Yujie Qian, Xiao Liu, Ming Ding, Jiezhong Qiu, Zhilin Yang, and Jie Tang. Glm: General language model pretraining with autoregressive blank infilling. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Paper...

    2022

  18. [26]

    Eva: Exploring the limits of masked visual represen- tation learning at scale

    Yuxin Fang, Wen Wang, Binhui Xie, Quan Sun, Ledell Wu, Xinggang Wang, Tiejun Huang, Xinlong Wang, and Yue Cao. Eva: Exploring the limits of masked visual represen- tation learning at scale. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (C...

    2023

  19. [27]

    Mme: A comprehensive evaluation benchmark for multimodal large language models

    Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Zhenyu Qiu, Wei Lin, Jin- rui Yang, Xiawu Zheng, Ke Li, Xing Sun, and Ron- grong Ji. Mme: A comprehensive evaluation benchmark for multimodal large language models. arXiv preprint arXiv:2306.13394, 2023. 2, 5

    2023 arXiv

  20. [28]

    A challenger to gpt-4v? early explorations of gemini in visual expertise

    Chaoyou Fu, Renrui Zhang, Zihan Wang, Yubo Huang, Zhengye Zhang, Longtian Qiu, Gaoxiang Ye, Yunhang Shen, Mengdan Zhang, Peixian Chen, Sirui Zhao, Shao- hui Lin, Deqiang Jiang, Di Yin, Peng Gao, Ke Li, Hong- sheng Li, and Xing Sun. A challenger to gpt-4v? early explorations of...

    2023

  21. [29]

    Zhang, Pan Lu, Conghui He, Xi- angyu Yue, Hongsheng Li, and Yu Jiao Qiao

    Peng Gao, Jiaming Han, Renrui Zhang, Ziyi Lin, Shijie Geng, Aojun Zhou, W. Zhang, Pan Lu, Conghui He, Xi- angyu Yue, Hongsheng Li, and Yu Jiao Qiao. Llama- adapter v2: Parameter-efficient visual instruction model. ArXiv, abs/2304.15010, 2023. 2

    2023 arXiv

  22. [30]

    Planting a seed of vision in large language model

    Yuying Ge, Yixiao Ge, Ziyun Zeng, Xintao Wang, and Ying Shan. Planting a seed of vision in large language model. 3

  23. [31]

    Hallusionbench: An advanced diagnostic suite for entangled language hallucination & visual illusion in large vision-language models, 2023

    Tianrui Guan, Fuxiao Liu, Xiyang Wu, Ruiqi Xian, Zongxia Li, Xiaoyu Liu, Xijun Wang, Lichang Chen, Furong Huang, Yaser Yacoob, Dinesh Manocha, and Tianyi Zhou. Hallusionbench: An advanced diagnostic suite for entangled language hallucination & visual illusion in large vision-l...

    2023

  24. [32]

    Wan- 8 juan: A comprehensive multimodal dataset for advancing english and chinese large models

    Conghui He, Zhenjiang Jin, Chaoxi Xu, Jiantao Qiu, Bin Wang, Wei Li, Hang Yan, Jiaqi Wang, and Da Lin. Wan- 8 juan: A comprehensive multimodal dataset for advancing english and chinese large models. ArXiv, abs/2308.10755,

  25. [33]

    LoRA: Low-rank adaptation of large language mod- els

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen- Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language mod- els. In International Conference on Learning Representa- tions, 2022. 2, 3

    2022

  26. [34]

    Opera: Alleviating hallucination in multi- modal large language models via over-trust penalty and retrospection-allocation

    Qidong Huang, Xiaoyi Dong, Pan Zhang, Bin Wang, Con- ghui He, Jiaqi Wang, Dahua Lin, Weiming Zhang, and Nenghai Yu. Opera: Alleviating hallucination in multi- modal large language models via over-trust penalty and retrospection-allocation. arXiv preprint arXiv:2311.17911,

  27. [35]

    Gqa: A new dataset for real-world visual reasoning and compositional question answering

    Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. Conference on Computer Vision and Pattern Recognition (CVPR), 2019. 4

    2019

  28. [36]

    Scaling up visual and vision-language representation learning with noisy text supervision

    Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In Pro- ceedings of the International Conference on Machine learn...

    2021

  29. [37]

    Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lam- ple, Lucile Saulnier, L ´elio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Tho...

    2023

  30. [38]

    Mixtral of experts

    Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Deven- dra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. Mixtral of experts. arXiv preprint arXiv:2401.04088, 2024. 2

    2024 arXiv

  31. [39]

    Dvqa: Understanding data visualizations via ques- tion answering

    Kushal Kafle, Brian Price, Scott Cohen, and Christopher Kanan. Dvqa: Understanding data visualizations via ques- tion answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5648–5656,

  32. [40]

    A diagram is worth a dozen images

    Aniruddha Kembhavi, Mike Salvato, Eric Kolve, Minjoon Seo, Hannaneh Hajishirzi, and Ali Farhadi. A diagram is worth a dozen images. In Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11–14, 2016, Proceedings, Part IV 14, pages 235–

    2016

  33. [41]

    Springer, 2016. 2, 4, 5

    2016

  34. [42]

    Seed-bench: Benchmarking multi- modal llms with generative comprehension, 2023

    Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, Yix- iao Ge, and Ying Shan. Seed-bench: Benchmarking multi- modal llms with generative comprehension, 2023. 2, 5

    2023

  35. [43]

    Otter: A multi-modal model with in-context instruction tuning

    Bo Li, Yuanhan Zhang, Liangyu Chen, Jinghao Wang, Jingkang Yang, and Ziwei Liu. Otter: A multi-modal model with in-context instruction tuning. arXiv.org, 2023. 3

    2023

  36. [44]

    Blip: Bootstrapping language-image pre-training for uni- fied vision-language understanding and generation

    Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre-training for uni- fied vision-language understanding and generation. In Pro- ceedings of the International Conference on Machine learn- ing (ICML), pages 12888–12900. PMLR, 2022. 3

    2022

  37. [45]

    Grounded language-image pre-training

    Liunian Harold Li*, Pengchuan Zhang*, Haotian Zhang*, Jianwei Yang, Chunyuan Li, Yiwu Zhong, Lijuan Wang, Lu Yuan, Lei Zhang, Jenq-Neng Hwang, Kai-Wei Chang, and Jianfeng Gao. Grounded language-image pre-training. In Proceedings of the IEEE/CVF Conference on Computer Vision an...

    2022

  38. [46]

    Evaluating object hallucination in large vision-language models, 2023

    Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision-language models, 2023. 2, 5

    2023

  39. [47]

    Monkey: Image resolution and text label are important things for large multi-modal models

    Zhang Li, Biao Yang, Qiang Liu, Zhiyin Ma, Shuo Zhang, Jingxu Yang, Yabo Sun, Yuliang Liu, and Xiang Bai. Monkey: Image resolution and text label are important things for large multi-modal models. arXiv preprint arXiv:2311.06607, 2023. 3

    2023

  40. [48]

    Mmc: Advancing multimodal chart understand- ing with large-scale instruction tuning

    Fuxiao Liu, Xiaoyang Wang, Wenlin Yao, Jianshu Chen, Kaiqiang Song, Sangwoo Cho, Yaser Yacoob, and Dong Yu. Mmc: Advancing multimodal chart understand- ing with large-scale instruction tuning. arXiv preprint arXiv:2311.10774, 2023. 4

    2023

  41. [49]

    Improved baselines with visual instruction tuning

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. arXiv preprint arXiv:2310.03744, 2023. 2

    2023 arXiv

  42. [50]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. arXiv.org, 2023. 2, 3, 4, 5

    2023

  43. [51]

    Grounding dino: Marrying dino with grounded pre-training for open-set object detection

    Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. arXiv.org, 2023. 3

    2023

  44. [52]

    Mm- bench: Is your multi-modal model an all-around player? arXiv:2307.06281, 2023

    Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhnag, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, Kai Chen, and Dahua Lin. Mm- bench: Is your multi-modal model an all-around player? arXiv:2307.06281, 2023. 2, 5

    2023 arXiv

  45. [53]

    Mathvista: Evaluating mathematical reasoning of foundation models in visual con- texts

    Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chun- yuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evaluating mathematical reasoning of foundation models in visual con- texts. In International Conference on Learning Represe...

    2024

  46. [54]

    Inter-gps: In- terpretable geometry problem solving with formal language and symbolic reasoning

    Pan Lu, Ran Gong, Shibiao Jiang, Liang Qiu, Siyuan Huang, Xiaodan Liang, and Song-Chun Zhu. Inter-gps: In- terpretable geometry problem solving with formal language and symbolic reasoning. In The 59th Annual Meeting of the Association for Computational Linguistics (ACL), 2021. 4

    2021

  47. [55]

    Learn to explain: Multimodal rea- soning via thought chains for science question answer- ing

    Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai- Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal rea- soning via thought chains for science question answer- ing. Advances in Neural Information Processing Systems , 35:2...

    2022

  48. [56]

    Ok-vqa: A visual question answering benchmark requiring external knowledge

    Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. Ok-vqa: A visual question answering benchmark requiring external knowledge. In Proceedings of the IEEE/cvf conference on computer vision and pattern recognition, pages 3195–3204, 2019. 4

    2019

  49. [57]

    Chartqa: A benchmark for question answering about charts with visual and logical reasoning

    Ahmed Masry, Do Xuan Long, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. Chartqa: A benchmark for question answering about charts with visual and logical reasoning. arXiv preprint arXiv:2203.10244, 2022. 2, 4, 5 9

    2022 arXiv

  50. [58]

    OpenAI. Chatgpt. https://openai.com/blog/ chatgpt, 2022. 2

    2022

  51. [59]

    Gpt-4 technical report, 2023

    OpenAI. Gpt-4 technical report, 2023. 1, 2, 3

    2023

  52. [60]

    Vicente Ordonez, Girish Kulkarni, and Tamara L. Berg. Im2text: Describing images using 1 million captioned photographs. In Neural Information Processing Systems (NIPS), 2011. 4

    2011

  53. [61]

    Training language models to follow instructions with human feed- back

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Car- roll Wainwright, Pamela Mishkin, Chong Zhang, Sand- hini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feed- back. Advances in Neural Information Processing Systems (N...

    2022

  54. [62]

    The refinedweb dataset for falcon llm: Outperforming curated corpora with web data, and web data only, 2023

    Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Alessandro Cappelli, Hamza Alobei- dli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Lau- nay. The refinedweb dataset for falcon llm: Outperforming curated corpora with web data, and web data only, 2023. 3

    2023

  55. [63]

    Kosmos-2: Grounding multimodal large language models to the world

    Zhiliang Peng, Wenhui Wang, Li Dong, Yaru Hao, Shaohan Huang, Shuming Ma, and Furu Wei. Kosmos-2: Grounding multimodal large language models to the world. arXiv.org,

  56. [64]

    Gpt4point: A unified framework for point-language under- standing and generation, 2023

    Zhangyang Qi, Ye Fang, Zeyi Sun, Xiaoyang Wu, Tong Wu, Jiaqi Wang, Dahua Lin, and Hengshuang Zhao. Gpt4point: A unified framework for point-language under- standing and generation, 2023. 3

    2023

  57. [65]

    Gemini vs gpt-4v: A preliminary comparison and combination of vision-language models through qualitative cases, 2023

    Zhangyang Qi, Ye Fang, Mengchen Zhang, Zeyi Sun, Tong Wu, Ziwei Liu, Dahua Lin, Jiaqi Wang, and Hengshuang Zhao. Gemini vs gpt-4v: A preliminary comparison and combination of vision-language models through qualitative cases, 2023. 3

    2023

  58. [66]

    Introducing qwen-7b: Open foundation and human- aligned models (of the state-of-the-arts), 2023

    Qwen. Introducing qwen-7b: Open foundation and human- aligned models (of the state-of-the-arts), 2023. 2

    2023

  59. [67]

    Learn- ing transferable visual models from natural language super- vision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learn- ing transferable visual models from natural language super- vision. In Proceedings of the International Conference ...

    2021

  60. [68]

    Improving language understanding by gen- erative pre-training

    Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. Improving language understanding by gen- erative pre-training. 2018. 2

    2018

  61. [69]

    Exploring the limits of transfer learning with a unified text-to-text transformer

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research (JMLR), 21(1):5485–5551, 2020. 2

    2020

  62. [70]

    Laion- 400m: Open dataset of clip-filtered 400 million image-text pairs

    Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. Laion- 400m: Open dataset of clip-filtered 400 million image-text pairs. arXiv preprint arXiv:2111.02114, 2021. 4

    2021 arXiv

  63. [71]

    A-okvqa: A benchmark for visual question answering using world knowledge

    Dustin Schwenk, Apoorv Khandelwal, Christopher Clark, Kenneth Marino, and Roozbeh Mottaghi. A-okvqa: A benchmark for visual question answering using world knowledge. In European Conference on Computer Vision, pages 146–162. Springer, 2022. 4

    2022

  64. [72]

    Kvqa: Knowledge-aware visual question answering

    Sanket Shah, Anand Mishra, Naganand Yadati, and Partha Pratim Talukdar. Kvqa: Knowledge-aware visual question answering. In Proceedings of the AAAI conference on artificial intelligence, 2019. 4

    2019

  65. [73]

    Conceptual captions: A cleaned, hypernymed, im- age alt-text dataset for automatic image captioning

    Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernymed, im- age alt-text dataset for automatic image captioning. In Pro- ceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Paper...

    2018

  66. [74]

    Textcaps: a dataset for image caption- ing with reading comprehension

    Oleksii Sidorov, Ronghang Hu, Marcus Rohrbach, and Amanpreet Singh. Textcaps: a dataset for image caption- ing with reading comprehension. In Computer Vision– ECCV 2020: 16th European Conference, Glasgow, UK, Au- gust 23–28, 2020, Proceedings, Part II 16, pages 742–758. Spring...

    2020

  67. [75]

    Generative pretraining in mul- timodality

    Quan Sun, Qiying Yu, Yufeng Cui, Fan Zhang, Xiaosong Zhang, Yueze Wang, Hongcheng Gao, Jingjing Liu, Tiejun Huang, and Xinlong Wang. Generative pretraining in mul- timodality. Jul 2023. 3

    2023

  68. [76]

    Alpha-CLIP: A clip model focusing on wherever you want

    Zeyi Sun, Ye Fang, Tong Wu, Pan Zhang, Yuhang Zang, Shu Kong, Yuanjun Xiong, Dahua Lin, and Jiaqi Wang. Alpha-CLIP: A clip model focusing on wherever you want. arXiv preprint arXiv:2312.03818, 2023. 3

    2023

  69. [77]

    Gemini: A family of highly capable multi- modal models, 2023

    Gemini Team. Gemini: A family of highly capable multi- modal models, 2023. 1, 2

    2023

  70. [78]

    Internlm: A multilingual language model with progressively enhanced capabilities

    InternLM Team. Internlm: A multilingual language model with progressively enhanced capabilities. https : / / github.com/InternLM/InternLM, 2023. 1, 2, 4

    2023

  71. [79]

    Llama: Open and efficient foundation language mod- els

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth ´ee Lacroix, Bap- tiste Rozi `ere, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language mod- els. arXiv.org, 2023. 2

    2023

  72. [80]

    Llama 2: Open foundation and fine-tuned chat models,

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models,

  73. [81]

    Vigc: Visual instruction generation and correc- tion

    Bin Wang, Fan Wu, Xiao Han, Jiahui Peng, Huaping Zhong, Pan Zhang, Xiaoyi Dong, Weijia Li, Wei Li, Jiaqi Wang, et al. Vigc: Visual instruction generation and correc- tion. arXiv.org, 2023. 3

    2023

  74. [82]

    To see is to believe: Prompting gpt-4v for better visual instruction tuning

    Junke Wang, Lingchen Meng, Zejia Weng, Bo He, Zuxuan Wu, and Yu-Gang Jiang. To see is to believe: Prompting gpt-4v for better visual instruction tuning. arXiv preprint arXiv:2311.07574, 2023. 4

    2023

  75. [83]

    Cogvlm: Visual expert for pretrained language models, 2023

    Weihan Wang, Qingsong Lv, Wenmeng Yu, Wenyi Hong, Ji Qi, Yan Wang, Junhui Ji, Zhuoyi Yang, Lei Zhao, Xixuan Song, Jiazheng Xu, Bin Xu, Juanzi Li, Yuxiao Dong, Ming Ding, and Jie Tang. Cogvlm: Visual expert for pretrained language models, 2023. 2, 3

    2023

  76. [84]

    Vary: Scaling up the vision vocab- ulary for large vision-language models

    Haoran Wei, Lingyu Kong, Jinyue Chen, Liang Zhao, Zheng Ge, Jinrong Yang, Jianjian Sun, Chunrui Han, and Xiangyu Zhang. Vary: Scaling up the vision vocab- ulary for large vision-language models. arXiv preprint arXiv:2312.06109, 2023. 3

    2023

  77. [85]

    Skywork: A more open bilingual foundation 10 model

    Tianwen Wei, Liang Zhao, Lichang Zhang, Bo Zhu, Lijie Wang, Haihua Yang, Biye Li, Cheng Cheng, Weiwei L ¨u, Rui Hu, et al. Skywork: A more open bilingual foundation 10 model. arXiv preprint arXiv:2310.19341, 2023. 3

    2023

  78. [86]

    Q-bench: A benchmark for general-purpose foundation models on low-level vision

    Haoning Wu, Zicheng Zhang, Erli Zhang, Chaofeng Chen, Liang Liao, Annan Wang, Chunyi Li, Wenxiu Sun, Qiong Yan, Guangtao Zhai, et al. Q-bench: A benchmark for general-purpose foundation models on low-level vision. arXiv preprint arXiv:2309.14181, 2023. 2, 5

    2023

  79. [87]

    mplug-owl: Modularization empowers large language models with multimodality.arXiv.org, 2023

    Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang, Anwen Hu, Pengcheng Shi, Yaya Shi, et al. mplug-owl: Modularization empowers large language models with multimodality.arXiv.org, 2023. 3

    2023

  80. [88]

    Woodpecker: Hallucination correc- tion for multimodal large language models

    Shukang Yin, Chaoyou Fu, Sirui Zhao, Tong Xu, Hao Wang, Dianbo Sui, Yunhang Shen, Ke Li, Xing Sun, and Enhong Chen. Woodpecker: Hallucination correc- tion for multimodal large language models. arXiv preprint arXiv:2310.16045, 2023. 3

    2023

  81. [89]

    From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions

    Peter Young, Alice Lai, Micah Hodosh, and Julia Hock- enmaier. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. Transactions of the Association for Computa- tional Linguistics, 2:67–78, 2014. 4

    2014

  82. [90]

    Scaling autoregressive multi-modal models: Pretraining and instruction tuning

    Lili Yu, Bowen Shi, Ramakanth Pasunuru, Benjamin Muller, Olga Golovneva, Tianlu Wang, Arun Babu, Binh Tang, Brian Karrer, Shelly Sheynin, Candace Ross, Adam Polyak, Russell Howes, Vasu Sharma, Puxin Xu, Hov- hannes Tamoyan, Oron Ashual, Uriel Singer, Shang-Wen Li, Susan Zhang,...

  83. [91]

    Mm-vet: Evaluating large multimodal models for inte- grated capabilities

    Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. Mm-vet: Evaluating large multimodal models for inte- grated capabilities. arXiv preprint arXiv:2308.02490, 2023. 2, 5

    2023 arXiv

  84. [92]

    Mmmu: A massive multi-discipline multi- modal understanding and reasoning benchmark for expert agi

    Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, Cong Wei, Botao Yu, Ruibin Yuan, Renliang Sun, Ming Yin, Boyuan Zheng, Zhenzhu Yang, Yibo Liu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. Mmmu: A m...

    2023 arXiv

  85. [93]

    Contextual object detection with multimodal large language models

    Yuhang Zang, Wei Li, Jun Han, Kaiyang Zhou, and Chen Change Loy. Contextual object detection with multimodal large language models. arXiv preprint arXiv:2305.18279, 2023. 3

    2023

  86. [94]

    GLM-130b: An open bilingual pre- trained model

    Aohan Zeng, Xiao Liu, Zhengxiao Du, Zihan Wang, Hanyu Lai, Ming Ding, Zhuoyi Yang, Yifan Xu, Wendi Zheng, Xiao Xia, Weng Lam Tam, Zixuan Ma, Yufei Xue, Jidong Zhai, Wenguang Chen, Zhiyuan Liu, Peng Zhang, Yuxiao Dong, and Jie Tang. GLM-130b: An open bilingual pre- trained mode...

    2023

  87. [95]

    Glipv2: Unifying localization and vision-language understanding

    Haotian Zhang, Pengchuan Zhang, Xiaowei Hu, Yen-Chun Chen, Liunian Li, Xiyang Dai, Lijuan Wang, Lu Yuan, Jenq-Neng Hwang, and Jianfeng Gao. Glipv2: Unifying localization and vision-language understanding. Advances in Neural Information Processing Systems (NeurIPS) , 35:36067–3...

    2022

  88. [96]

    Internlm- xcomposer: A vision-language large model for advanced text-image comprehension and composition

    Pan Zhang, Xiaoyi Dong Bin Wang, Yuhang Cao, Chao Xu, Linke Ouyang, Zhiyuan Zhao, Shuangrui Ding, Songyang Zhang, Haodong Duan, Hang Yan, et al. Internlm- xcomposer: A vision-language large model for advanced text-image comprehension and composition. arXiv preprint arXiv:2309....

    2023

  89. [97]

    OPT: Open pre-trained transformer language models

    Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. OPT: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068, 2022. 2

    2022 arXiv

  90. [98]

    Mmicl: Empowering vision-language model with multi-modal in-context learning

    Haozhe Zhao, Zefan Cai, Shuzheng Si, Xiaojian Ma, Kaikai An, Liang Chen, Zixuan Liu, Sheng Wang, Wenjuan Han, and Baobao Chang. Mmicl: Empowering vision-language model with multi-modal in-context learning. arXiv.org,

  91. [99]

    Mllm-dataengine: An iterative refinement approach for mllm

    Zhiyuan Zhao, Linke Ouyang, Bin Wang, Siyuan Huang, Pan Zhang, Xiaoyi Dong, Jiaqi Wang, and Conghui He. Mllm-dataengine: An iterative refinement approach for mllm. arXiv.org, 2023. 3

    2023

  92. [100]

    Beyond hallucinations: En- hancing lvlms through hallucination-aware direct prefer- ence optimization

    Zhiyuan Zhao, Bin Wang, Linke Ouyang, Xiaoyi Dong, Ji- aqi Wang, and Conghui He. Beyond hallucinations: En- hancing lvlms through hallucination-aware direct prefer- ence optimization. arXiv preprint arXiv:2311.16839, 2023. 3

    2023

  93. [101]

    中国水墨画：流动的诗意与东方美学

    Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: Enhancing vision- language understanding with advanced large language models. arXiv.org, 2023. 2, 3 11 French Pastries: A Sweet Indulgence French pastries are something that everyone should indulge...

    2023

  94. [102]

    The first part should cover the swimming equipments

    Head position Please write an blog based on the title: how to learn swim. The first part should cover the swimming equipments. The second part should discuss warm-up exercises before swimming. The last part tells us how to swim, including different swimming techniques. For tho...

  95. [103]

    The push- forward stage refers to the time when the forearm enters the water, moves downward, and extends forward until the fingertips pass the hips

    Arm movement Tips for Visiting Shanghai Arm movement is divided into two stages, namely the push-forward stage and the pull-back stage. The push- forward stage refers to the time when the forearm enters the water, moves downward, and extends forward until the fingertips pass t...

  96. [104]

    The park's natural beauty and geothermal wonders are truly awe-inspiring

    Leg movement Yellowstone National Park is a must-see destination for any traveler. The park's natural beauty and geothermal wonders are truly awe-inspiring. From the stunning Old Faithful geyser to the breathtaking Grand Canyon of Yellowstone, there is no shortage of amazing s...

  97. [105]

    ⽉⽛泉”。每到夜晚，⽉亮升起时，⽉⽛泉周围会发出阵阵清脆的响声，犹如天籁之⾳， 令⼈⼼旷神怡。 请告诉我图中代码的功能是什么？ 这段代码是 Python 语言的一个函数，名为 bubble_sort。 它使用了冒泡排序算法来对给定的列表进行排序。该算法通过不断交换相邻的元素，将较大的数逐渐“浮

    Breathing method When swimming, turn your head sideways to breathe in natural rhythm, and look at the side about 60cm away from your face. You can also exhale while swimming in shallow water to control the air flow rate of the mouth and nose. Back crawl: The main difference be...

    1974

Pith tools

Reviewed May 17, 2026 · model on record in the stance chip above.