Pith. sign in

REVIEW 4 major objections 4 minor 50 references

Region-Level Context-Aware Multimodal Understanding

T0 review · 4 major / 4 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A training format that places each object's text next to its bounding box lets multimodal models associate what they see with what they know about an object.

desk verdict New task and resources for region-level context in MLLMs, but the main capability claim needs a leakage check and text-only baseline before it lands. read the letter →

arxiv 2508.12263 v2 pith:QVYLTILM submitted 2025-08-17 cs.CV cs.AI

Hongliang Wei , Xianqi Zhang , Xingtao Wang , Xiaopeng Fan , Debin Zhao This is my paper
classification cs.CVcs.AI
keywords multimodallargelanguagemodelsregion-levelcontext-awareunderstandingvisualinstructiontuningpersonalizedreferringexpressiongenerationretrieval-augmentedreference-freeevaluationboundingboxgrounding
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Multimodal large language models can describe images, but they have no built-in way to use separately supplied facts about a specific object in the picture — say, that the dog on the left is named Bella and was adopted in 2022. This paper defines that missing ability as Region-level Context-aware Multimodal Understanding (RCMU) and claims it can be taught by a simple instruction-tuning format: write each object's text immediately after its bounding box coordinates, and train the model on questions that cross from an object's looks to its profile and back. The authors build a large automatically constructed dataset for this, fine-tune open-weight vision-language models with LoRA, and report that the resulting tuned models beat their baselines by wide margins on region-level question answering, contextual captioning, and citation generation, and that the skill transfers without extra training to personalized visual question answering. If the claim holds, region-level context awareness is not a new architecture but a reusable training format that gives ordinary open-weight models a practical capability for personalized assistants and multimodal retrieval.

What carries the argument

The load-bearing object is the RCVIT input template: the image token, then for each object a line of the form 'The <object> located at <[x1,y1,x2,y2]> in the image: <object information>', then the user instruction. The bounding-box coordinates are the associational hinge: they give the model a spatial anchor to bind each textual profile to the corresponding visual content during fine-tuning, and the same coordinate-anchored format at inference lets the model ground new text to new regions. A secondary piece of machinery, the reference-free RCIDScore metric, evaluates context-aware captions by averaging contextual coverage, contextual accuracy, and context-visual consistency with a CLIP-style

What would settle it

Take the RCMU context-to-visual and visual-to-context test questions and run them in text-only mode — keep the profiles and questions, drop the image. If a tuned model still scores near its full multimodal accuracy, the profiles leak the visual answers and the claimed region-level association is not driving the results. A manual audit of a sample of profiles for appearance-related words would give the same verdict.

Watch

Extended reading notes

Core claim

The central discovery is that bounding box coordinates act as the binding key between visual and textual information. In the RCVIT prompt, each object's textual profile is presented as the object located at a bounding box with its information following; fine-tuning the model to answer questions that require crossing from the visual to the textual side and back teaches it to attach the profile to the correct region. After training, the model handles unseen images and objects in the same format, answering visual-to-context questions (locate an object by sight, then report its profile), context-to-visual questions (locate an object by profile, then describe its appearance), and unanswerable que

Load-bearing premise

The pipeline's language model must generate textual profiles that contain no visual information about the objects; if a profile leaks appearance traits such as colors, clothing, or pose, then context-to-visual questions can be solved by reading text alone, and the reported cross-modal binding is not what the experiments demonstrate.

Editorial extensions

If this is right

  • A small open-weight model tuned this way can exceed its base model by more than fifty points on region-level VQA, narrowing the gap to much larger proprietary systems.
  • The region-level capability transfers to multimodal retrieval-augmented generation and personalized conversation without per-user fine-tuning, unlike approaches that require training on each user's data.
  • Training with a roughly balanced mix of answerable and unanswerable questions lets the model explicitly refuse when the supplied context lacks the answer, a useful reliability property for grounded assistants.
  • The whole recipe runs with LoRA on two consumer GPUs, suggesting the capability is a matter of prompt format and data rather than model scale.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper leaves untested whether the same format would work on a base model without pretrained familiarity with bounding-box tokens; if not, RCVIT is better described as activating latent grounding skills than as supplying entirely new ones.
  • A text-only control — presenting the same context-to-visual questions with the image removed — would test whether the generated profiles actually avoid visual leakage; high text-only accuracy would weaken the cross-modal reading of the reported gains.
  • The bounding-box binding idea generalizes beyond vision: timestamps could play the same role for video or audio events, and entity identifiers could bind text to database rows, turning RCMU into a general entity-level context-awareness recipe.
  • The contextual components of RCIDScore could double as a reward signal for reinforcement learning during generation, which might recover the slight drop in visual caption quality the paper reports for its larger tuned model.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper introduces Region-level Context-aware Multimodal Understanding (RCMU), a task requiring MLLMs to answer instructions using both visual content and textual information attached to object bounding boxes. It proposes RCVIT, an instruction-tuning format in which object-level text is prefixed with box coordinates; an automated pipeline that builds a large RCMU dataset from REG data; a new benchmark (RC&P-Bench) for personalized multimodal understanding; and RCIDScore, a reference-free metric for contextualized image descriptions. The authors fine-tune Qwen2-VL 2B/7B with LoRA on the RCMU dataset (RC-Qwen2-VL) and report large gains over base models on RCMU tasks, on RC&P-Bench, and on the Yo'LLaVA personalized QA benchmark, along with applications to multimodal RAG and personalized conversation.

Significance. If the reported capability is real, the paper is a useful contribution: it defines a plausible new task family, releases a large dataset and trained models, and introduces a new benchmark and metric. The release of data, models, and benchmark is a concrete strength. The paper also includes ablation studies on citation annotations and unanswerable-data ratios, and it evaluates on the external Yo'LLaVA benchmark. However, the central evidence is currently self-contained: the authors define the task, generate the training set, build the test set, propose the evaluation metric, and show that their own models win. The load-bearing claim that RCVIT's bounding-box mechanism enables cross-modal association is not yet isolated from simpler explanations, and the reported gains on context-to-visual questions could be inflated by textual leakage or by the absence of controlled baselines. The concerns below are specific and fixable with additional experiments.

major comments (4)
  1. [Section III-B, Table II] Potential visual-attribute leakage in textual contexts is not tested. The pipeline instructs GPT-4o to 'avoid generating any visual information about the objects,' but no verification is reported. If the generated contexts mention colors, shapes, clothing, or other visual properties, then Context-to-Visual (C2V) questions can be answered from the text alone, and the +24 to +41 point C2V gains in Table II would not demonstrate cross-modal association. This concern also applies to the GPT-4o-generated personalized information in RC&P-Bench (Section IV). I request two additions: (1) a leakage audit, e.g., sample N generated contexts and have human annotators or a VQA model check whether visual attributes needed by C2V questions appear in the text; (2) a text-only baseline on the RCMU test set, where the same context and question are given without the image. This directly tests whether the i
  2. [Section III-C, Table II] The design of RCVIT is not ablated. The claimed mechanism is that bounding-box coordinates allow the model to associate object text with visual regions, but the comparison in Table II is between RC-Qwen2-VL (fine-tuned on RCMU data) and the original Qwen2-VL (not fine-tuned on RCMU data). The large gains may therefore reflect general instruction tuning on a new dataset rather than the specific coordinate-grounding format. I ask for a training-format ablation on the same RCMU data: (i) full RCVIT format, (ii) textual context without bounding-box tokens, (iii) bounding-box tokens without textual context, and ideally (iv) in-context examples instead of fine-tuning. Without this, the central claim that 'the model utilizes bounding box coordinates to effectively associate objects' is not directly supported. Additionally, the paper should state whether baseline models in Table II received exac
  3. [Section V-B, Tables II and V] The main quantitative results rely on LLM-as-judge evaluation without a reported human-agreement study. RCVQA answers are judged by DeepSeek V3, and citation support is judged by Gemini Flash 1.5. Because the test set is self-constructed and includes many unanswerable questions, the judge may reward wording or format artifacts rather than semantic correctness. I request a human agreement study on a random sample (e.g., 100–200 responses per task) reporting Cohen's kappa or similar between the LLM judge and human raters, plus qualitative examples of judge errors. This is important because the headline improvements in Table II are so large that even a small judge bias could change the ranking.
  4. [Section IV, Table III] The RC&P-Bench results need clarification on retrieval and input format. The paper says it follows RAP and employs a multimodal retriever, but it does not specify whether all baselines were given the same reference images, personalized text, and question images. RAP-LLaVA's near-random scores (4.97 overall Closed-World, 6.13 Open-World, 7.27 Oracle) are suspicious and suggest the baseline may not have received a comparable setup. Also, because the personalized information for object entities is generated by GPT-4o, the same leakage concern as in Section III-B applies to the C2V questions in Table III. I ask the authors to report the exact prompt/retrieval setup for each baseline and to include a text-only ablation on RC&P-Bench.
minor comments (4)
  1. [Table I] The row 'Average number of citations per description' says 55.69, which is identical to the average description length in words and is numerically implausible (a description cannot have 55.69 citations when it has roughly 2.29 objects per image). This is likely a typo and should be corrected.
  2. [Section III-B] There is a typo in the test-set description: 'we annotated retion-level context-aware image descriptions' should be 'region-level.' The header 'approach III-C' in Section III also appears to be missing an opening parenthesis.
  3. [Section III-D] RCIDScore is defined as the harmonic mean of RCIDScore_ctx and an LLM2CLIP-based CLIPScore, but the two components operate on very different scales and the CLIPScore values in Table II are much higher than the contextual sub-scores. It would help to report the distribution of the two components and to justify the harmonic-mean aggregation beyond a single human-correlation number.
  4. [Section V-C] The 'Closed-World' setting is said to assess overall system accuracy 'regardless of whether the correct information was retrieved.' This definition is ambiguous: if retrieval returns the wrong entity's profile, is the answer scored as correct only when the model detects the mismatch and says 'not enough information'? Please clarify the scoring rule for closed-world and open-world settings.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the central claim is an empirical tuning result with disjoint test data and external benchmarks; no equation-level reduction to inputs.

full rationale

The paper's central chain is empirical rather than derivational: it defines the RCMU task, constructs the RCMU dataset (Section III-B), proposes the RCVIT loss (Eq. 2, standard conditional log-likelihood), trains RC-Qwen2-VL, and evaluates on a manually annotated test set whose images and object information are explicitly disjoint from training (Section III-B), plus the external Yo'LLaVA personalized QA benchmark (Table IV). No equation-level reduction to a fitted parameter or self-defined target was found. The RCIDScore metric is validated against human judgments (Table VIII), providing independent support for its use. The only overlapping-author citation is [20] (Vary) in Related Work as a general mention of multiple vision encoders; it is not load-bearing. Flagged as a validity risk, not a circularity: the paper does not report a leakage check on the GPT-4o-generated textual contexts (Section III-B), and no text-only baseline is reported on the RCMU test set, so a confound in the C2V gains cannot be excluded. However, that is a missing control, not a circular derivation. Constructing one's own task, data, metric, and benchmark does not by itself make the evaluation circular; the reported gains are contingent empirical results, and the external benchmark and human-correlation study provide independent checks.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The paper makes no first-principles derivations and introduces no physical or mathematical entities. Its central method rests on the three domain assumptions above. No numbers are fitted to data, so there are no free parameters to charge.

assumptions (3)
  • domain assumption GPT-4o-generated personalized text contains no visual information about objects.
    Section III-B 'Textual Context Generation': the pipeline explicitly instructs GPT-4o to avoid generating visual information. Context-to-visual QA validity depends on this separation.
  • domain assumption MLLMs (e.g., Qwen2-VL) can interpret bounding box tokens in the text input and associate them with visual regions.
    RCVIT input format in Section III-C assumes the model can use coordinates; Qwen2-VL's grounding capability is inherited from pretraining and not verified here.
  • domain assumption LLM judges (DeepSeek V3 for RCVQA, Gemini Flash for citations) provide accurate assessments.
    Section V-B uses LLM-as-judge without reporting human agreement for these specific judgments.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Region-Level Context-Aware Multimodal Understanding." pith.science (2026). https://pith.science/paper/QVYLTILM

@misc{pith2026250812263,
  author       = {Pith},
  title        = {Pith review of: Region-Level Context-Aware Multimodal Understanding},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QVYLTILM}},
  note         = {Machine review of arXiv:2508.12263}
}
read the original abstract

Despite significant progress, existing research on Multimodal Large Language Models (MLLMs) mainly focuses on general visual understanding, overlooking the ability to integrate textual context associated with objects for a more context-aware multimodal understanding -- an ability we refer to as Region-level Context-aware Multimodal Understanding (RCMU). To address this limitation, we first formulate the RCMU task, which requires models to respond to user instructions by integrating both image content and textual information of regions or objects. To equip MLLMs with RCMU capabilities, we propose Region-level Context-aware Visual Instruction Tuning (RCVIT), which incorporates object information into the model input and enables the model to utilize bounding box coordinates to effectively associate objects' visual content with their textual information. To address the lack of datasets, we introduce the RCMU dataset, a large-scale visual instruction tuning dataset that covers multiple RCMU tasks. We also propose RC\&P-Bench, a comprehensive benchmark that can evaluate the performance of MLLMs in RCMU and multimodal personalized understanding tasks. Additionally, we propose a reference-free evaluation metric to perform a comprehensive and fine-grained evaluation of the region-level context-aware image descriptions. By performing RCVIT on Qwen2-VL models with the RCMU dataset, we developed RC-Qwen2-VL models. Experimental results indicate that RC-Qwen2-VL models not only achieve outstanding performance on multiple RCMU tasks but also demonstrate successful applications in multimodal RAG and personalized conversation. Our data, model and benchmark are available at https://github.com/hongliang-wei/RC-MLLM

Figures

Figures reproduced from arXiv: 2508.12263 by the authors.

Figure 1
Figure 1. Illustration of the RCMU Task. RCMU requires [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Responses of RC-Qwen2-VL 7B and Qwen2-VL 7B on RCMU tasks. RC-Qwen2-VL performs better on RCMU tasks, [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. Automatic data construction pipeline for the RCMU dataset. This pipeline can generate high-quality training data for [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: The RCIDScore evaluates contextualized image de [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: Illustrative Example from RC&P-Bench IV. RC&P-BENCH RC&P-Bench contains 3260 multiple-choice questions, cov￾ering 259 entities.Each question image contains at least two personalized entities, and all personalized entities within a single image belong to the same catego…
Figure 6
Figure 6. Figure 6: Showcase of our models in practical applica [PITH_FULL_IMAGE:figures/full_fig_p009_6.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

50 extracted references · 29 canonical work pages

  1. [1]

    Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context,

    M. Reid, N. Savinov, D. Teplyashin, D. Lepikhin, T. P. Lillicrap, J.-B. Alayrac, R. Soricut, A. Lazaridou, O. Firat, J. Schrittwieser, I. Antonoglou, R. Anil, S. Borgeaud, A. M. Dai, K. Millican, E. Dyer, M. Glaese, T. Sottiaux, B. jamin Lee, F. Viola, M. Reynolds, Y . Xu, J. Molloy, J. Chen, M. Isard, P. Barham, T. Hennigan, R. McIlroy, M. Johnson, J. Sc...

    arXiv 2024

  2. [2]

    Flamingo: a visual language model for few-shot learning,

    J.-B. Alayrac, J. Donahue, P. Luc, A. Miech, I. Barr, Y . Hasson, K. Lenc, A. Mensch, K. Millican, M. Reynolds, R. Ring, E. Rutherford, S. Cabi, T. Han, Z. Gong, S. Samangooei, M. Monteiro, J. Menick, S. Borgeaud, A. Brock, A. Nematzadeh, S. Sharifzadeh, M. Binkowski, R. Barreira, O. Vinyals, A. Zisserman, and K. Simonyan, “Flamingo: a visual language mod...

    arXiv

  3. [3]

    Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution,

    P. Wang, S. Bai, S. Tan, S. Wang, Z. Fan, J. Bai, K.-Y . Chen, X. Liu, J. Wang, W. Ge, Y . Fan, K. Dang, M. Du, X. Ren, R. Men, D. Liu, C. Zhou, J. Zhou, and J. Lin, “Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution,” ArXiv, vol. abs/2409.12191, 2024. [Online]. Available: https://api.semanticscholar.org/CorpusID:272704132

    arXiv 2024

  4. [5]

    Instructblip: Towards general-purpose vision- language models with instruction tuning,

    W. Dai, J. Li, D. Li, A. M. H. Tiong, J. Zhao, W. Wang, B. A. Li, P. Fung, and S. C. H. Hoi, “Instructblip: Towards general-purpose vision- language models with instruction tuning,” ArXiv, vol. abs/2305.06500,

    arXiv

  5. [6]

    Deepseek-vl: Towards real-world vision-language understanding,

    H. Lu, W. Liu, B. Zhang, B.-L. Wang, K. Dong, B. L. B. Liu), J. Sun, T. Ren, Z. Li, H. Yang, Y . Sun, C. Deng, H. Xu, Z. Xie, and C. Ruan, “Deepseek-vl: Towards real-world vision-language understanding,” ArXiv, vol. abs/2403.05525, 2024. [Online]. Available: https://api.semanticscholar.org/CorpusID:268297008

    arXiv 2024

  6. [7]

    Murag: Multimodal retrieval-augmented generator for open question answering over images and text,

    W. Chen, H. Hu, X. Chen, P. Verga, and W. W. Cohen, “Murag: Multimodal retrieval-augmented generator for open question answering over images and text,” ArXiv, vol. abs/2210.02928, 2022. [Online]. Available: https://api.semanticscholar.org/CorpusID:252735160

    arXiv 2022

  7. [8]

    Mmicl: Empowering vision-language model with multi- modal in-context learning,

    H. Zhao, Z. Cai, S. Si, X. Ma, K. An, L. Chen, Z. Liu, S. Wang, W. Han, and B. Chang, “Mmicl: Empowering vision-language model with multi- modal in-context learning,” ArXiv, vol. abs/2309.07915, 2023. [Online]. Available: https://api.semanticscholar.org/CorpusID:261823391

    arXiv 2023

  8. [9]

    CaMML: Context-Aware Multimodal Learner for Large Models

    Y . Chen, S. Zhang, B. Han, T. He, and B. Li, “Camml: Context-aware multimodal learner for large models,” ArXiv, vol. abs/2401.03149,

    work page Pith review arXiv

Show all 50 references
  1. [10]

    Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models,

    B. A. Plummer, L. Wang, C. M. Cervantes, J. C. Caicedo, J. Hocken- maier, and S. Lazebnik, “Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models,” in Proceedings of the IEEE international conference on computer vision , 2015, pp. ...

    2015

  2. [11]

    Referitgame: Referring to objects in photographs of natural scenes,

    S. Kazemzadeh, V . Ordonez, M. Matten, and T. Berg, “Referitgame: Referring to objects in photographs of natural scenes,” in Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), 2014, pp. 787–798

    2014

  3. [12]

    Panoptic scene graph generation,

    J. Yang, Y . Z. Ang, Z. Guo, K. Zhou, W. Zhang, and Z. Liu, “Panoptic scene graph generation,” in European Conference on Computer Vision . Springer, 2022, pp. 178–196

    2022

  4. [13]

    Glamm: Pixel grounding large multimodal model,

    H. Rasheed, M. Maaz, S. Shaji, A. Shaker, S. Khan, H. Cholakkal, R. M. Anwer, E. Xing, M.-H. Yang, and F. S. Khan, “Glamm: Pixel grounding large multimodal model,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2024, pp. 13 009–13 018

    2024

  5. [14]

    Bleu: a method for automatic evaluation of machine translation,

    K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu, “Bleu: a method for automatic evaluation of machine translation,” in Proceedings of the 40th annual meeting of the Association for Computational Linguistics , 2002, pp. 311–318

    2002

  6. [15]

    Rouge: A package for automatic evaluation of summaries,

    C.-Y . Lin, “Rouge: A package for automatic evaluation of summaries,” in Text summarization branches out , 2004, pp. 74–81

    2004

  7. [16]

    Cider: Consensus- based image description evaluation,

    R. Vedantam, C. Lawrence Zitnick, and D. Parikh, “Cider: Consensus- based image description evaluation,” in Proceedings of the IEEE confer- ence on computer vision and pattern recognition , 2015, pp. 4566–4575

    2015

  8. [17]

    Spice: Semantic propositional image caption evaluation,

    P. Anderson, B. Fernando, M. Johnson, and S. Gould, “Spice: Semantic propositional image caption evaluation,” in Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part V 14 . Springer, 2016, pp. 382–398

    2016

  9. [18]

    Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models,

    J. Li, D. Li, S. Savarese, and S. C. H. Hoi, “Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models,” in International Conference on Machine Learning ,

  10. [19]

    Visual instruction tuning,

    H. Liu, C. Li, Q. Wu, and Y . J. Lee, “Visual instruction tuning,” ArXiv, vol. abs/2304.08485, 2023. [Online]. Available: https://api.semanticscholar.org/CorpusID:258179774

    2023 arXiv

  11. [20]

    Vary: Scaling up the vision vocabulary for large vision-language models,

    H. Wei, L. Kong, J. Chen, L. Zhao, Z. Ge, J. Yang, J. Sun, C. Han, and X. Zhang, “Vary: Scaling up the vision vocabulary for large vision-language models,” ArXiv, vol. abs/2312.06109, 2023. [Online]. Available: https://api.semanticscholar.org/CorpusID:266163175

    2023 arXiv

  12. [21]

    Ferret-v2: An improved baseline for referring and grounding with large language models,

    H. Zhang, H. You, P. Dufter, B. Zhang, C. Chen, H.-Y . Chen, T.-J. Fu, W. Y . Wang, S.-F. Chang, Z. Gan, and Y . Yang, “Ferret-v2: An improved baseline for referring and grounding with large language models,” ArXiv, vol. abs/2404.07973, 2024. [Online]. Available: https://api.s...

    2024 arXiv

  13. [22]

    Available: https://api.semanticscholar.org/CorpusID: 256390509

    [Online]. Available: https://api.semanticscholar.org/CorpusID: 256390509

  14. [23]

    Mmict: Boosting multi-modal fine-tuning with in-context examples,

    T. Chen, E. Zhang, Y . Gao, K. Li, X. Sun, Y . Zhang, H. Li, and R. Ji, “Mmict: Boosting multi-modal fine-tuning with in-context examples,” ACM Transactions on Multimedia Computing, Communications and Applications

  15. [24]

    Learn to explain: Multimodal reasoning via thought chains for science question answering,

    P. Lu, S. Mishra, T. Xia, L. Qiu, K.-W. Chang, S.-C. Zhu, O. Tafjord, P. Clark, and A. Kalyan, “Learn to explain: Multimodal reasoning via thought chains for science question answering,” Advances in Neural Information Processing Systems , vol. 35, pp. 2507–2521, 2022

    2022

  16. [25]

    Cantor: Inspiring multimodal chain- of-thought of mllm,

    T. Gao, P. Chen, M. Zhang, C. Fu, Y . Shen, Y . Zhang, S. Zhang, X. Zheng, X. Sun, L. Cao et al. , “Cantor: Inspiring multimodal chain- of-thought of mllm,” in Proceedings of the 32nd ACM International Conference on Multimedia , 2024, pp. 9096–9105

    2024

  17. [26]

    Inf- llava: Dual-perspective perception for high-resolution multimodal large language model,

    Y . Ma, Z. Wang, X. Sun, W. Lin, Q. Zhou, J. Ji, and R. Ji, “Inf- llava: Dual-perspective perception for high-resolution multimodal large language model,” arXiv preprint arXiv:2407.16198 , 2024

    2024 arXiv

  18. [27]

    Video-llama: An instruction-tuned audio-visual language model for video understanding,

    H. Zhang, X. Li, and L. Bing, “Video-llama: An instruction-tuned audio-visual language model for video understanding,” in Conference on Empirical Methods in Natural Language Processing , 2023. [Online]. Available: https://api.semanticscholar.org/CorpusID:259075356

    2023

  19. [28]

    Video-rag: Visually-aligned retrieval-augmented long video comprehension,

    Y . Luo, X. Zheng, X. Yang, G. Li, H. Lin, J. Huang, J. Ji, F. Chao, J. Luo, and R. Ji, “Video-rag: Visually-aligned retrieval-augmented long video comprehension,” arXiv preprint arXiv:2411.13093 , 2024

    2024

  20. [29]

    Llava-next-interleave: Tackling multi-image, video, and 3d in large multimodal models,

    F. Li, R. Zhang, H. Zhang, Y . Zhang, B. Li, W. Li, Z. Ma, and C. Li, “Llava-next-interleave: Tackling multi-image, video, and 3d in large multimodal models,” ArXiv, vol. abs/2407.07895, 2024. [Online]. Available: https://api.semanticscholar.org/CorpusID:271088459

    2024 arXiv

  21. [30]

    Video-llava: Learning united visual representation by alignment before projection,

    B. Lin, B. Zhu, Y . Ye, M. Ning, P. Jin, and L. Yuan, “Video-llava: Learning united visual representation by alignment before projection,” in Conference on Empirical Methods in Natural Language Processing ,

  22. [31]

    Available: https://api.semanticscholar.org/CorpusID: 265281544

    [Online]. Available: https://api.semanticscholar.org/CorpusID: 265281544

  23. [32]

    Manipllm: Embodied multimodal large language model for object-centric robotic manipulation,

    X. Li, M. Zhang, Y . Geng, H. Geng, Y . Long, Y . Shen, R. Zhang, J. Liu, and H. Dong, “Manipllm: Embodied multimodal large language model for object-centric robotic manipulation,” 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pp. 18 061–18 070,

    2024

  24. [33]

    Rap: Retrieval-augmented personalization for multimodal large language models,

    H. Hao, J. Han, C. Li, Y .-F. Li, and X. Yue, “Rap: Retrieval-augmented personalization for multimodal large language models,” in Proceedings of the Computer Vision and Pattern Recognition Conference , 2025, pp. 14 538–14 548

    2025

  25. [34]

    Yo’llava: Your personalized language and vision assistant,

    T. Nguyen, H. Liu, Y . Li, M. Cai, U. Ojha, and Y . J. Lee, “Yo’llava: Your personalized language and vision assistant,” Advances in Neural Information Processing Systems , vol. 37, pp. 40 913–40 951, 2024

    2024

  26. [35]

    Jm3d & jm3d- llm: Elevating 3d representation with joint multi-modal cues,

    J. Ji, H. Wang, C. Wu, Y . Ma, X. Sun, and R. Ji, “Jm3d & jm3d- llm: Elevating 3d representation with joint multi-modal cues,” IEEE Transactions on Pattern Analysis and Machine Intelligence , 2024. 12

    2024

  27. [36]

    Palm-e: An embodied multimodal language model,

    D. Driess, F. Xia, M. S. M. Sajjadi, C. Lynch, A. Chowdhery, B. Ichter, A. Wahid, J. Tompson, Q. H. Vuong, T. Yu, W. Huang, Y . Chebotar, P. Sermanet, D. Duckworth, S. Levine, V . Vanhoucke, K. Hausman, M. Toussaint, K. Greff, A. Zeng, I. Mordatch, and P. R. Florence, “Palm-e:...

    2023

  28. [37]

    Llm2clip: Powerful language model unlock richer visual representation,

    W. Huang, A. Wu, Y . Yang, X. Luo, Y . Yang, L. Hu, Q. Dai, X. Dai, D. Chen, C. Luo et al. , “Llm2clip: Powerful language model unlock richer visual representation,” arXiv preprint arXiv:2411.04997 , 2024

    2024

  29. [38]

    Available: https://api.semanticscholar.org/CorpusID: 266573457

    [Online]. Available: https://api.semanticscholar.org/CorpusID: 266573457

  30. [39]

    Gpt-4o system card,

    A. Hurst, A. Lerer, A. P. Goucher, A. Perelman, A. Ramesh, A. Clark, A. Ostrow, A. Welihinda, A. Hayes, A. Radford et al. , “Gpt-4o system card,” arXiv preprint arXiv:2410.21276 , 2024

    2024 arXiv

  31. [40]

    How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites,

    Z. Chen, W. Wang, H. Tian, S. Ye, Z. Gao, E. Cui, W. Tong, K. Hu, J. Luo, Z. Ma et al. , “How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites,” arXiv preprint arXiv:2404.16821, 2024

    2024 arXiv

  32. [41]

    Myvlm: Personalizing vlms for user-specific queries,

    Y . Alaluf, E. Richardson, S. Tulyakov, K. Aberman, and D. Cohen- Or, “Myvlm: Personalizing vlms for user-specific queries,” in European Conference on Computer Vision . Springer, 2024, pp. 73–91

    2024

  33. [42]

    Clipscore: A reference-free evaluation metric for image captioning,

    J. Hessel, A. Holtzman, M. Forbes, R. L. Bras, and Y . Choi, “Clipscore: A reference-free evaluation metric for image captioning,” arXiv preprint arXiv:2104.08718, 2021

    2021 arXiv

  34. [43]

    Deepseek-v3 technical report,

    A. Liu, B. Feng, B. Xue, B. Wang, B. Wu, C. Lu, C. Zhao, C. Deng, C. Zhang, C. Ruan et al., “Deepseek-v3 technical report,” arXiv preprint arXiv:2412.19437, 2024

    2024 arXiv

  35. [44]

    Gemini 2.0: A new era of multimodal models,

    DeepMind, “Gemini 2.0: A new era of multimodal models,” Google DeepMind Research Blog, 2024. [Online]. Available: https://www. deepmind.com/blog/gemini-2-0-a-new-era-of-multimodal-models

    2024

  36. [45]

    Gem- ini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities,

    G. Comanici, E. Bieber, M. Schaekermann, I. Pasupat, N. Sachdeva, I. Dhillon, M. Blistein, O. Ram, D. Zhang, E. Rosen et al. , “Gem- ini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities,” arXiv preprint ar...

    2025 arXiv

  37. [47]

    Intern vl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks,

    Z. Chen, J. Wu, W. Wang, W. Su, G. Chen, S. Xing, Z. Muyan, Q. Zhang, X. Zhu, L. Lu, B. Li, P. Luo, T. Lu, Y . Qiao, and J. Dai, “Intern vl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks,” 2024 IEEE/CVF Conference on Computer Vision and P...

    2024

  38. [48]

    Minicpm-v: A gpt-4v level mllm on your phone,

    Y . Yao, T. Yu, A. Zhang, C. Wang, J. Cui, H. Zhu, T. Cai, H. Li, W. Zhao, Z. He et al. , “Minicpm-v: A gpt-4v level mllm on your phone,” arXiv preprint arXiv:2408.01800, 2024

    2024 arXiv

  39. [50]

    Enabling large language models to generate text with citations,

    T. Gao, H. Yen, J. Yu, and D. Chen, “Enabling large language models to generate text with citations,” arXiv preprint arXiv:2305.14627 , 2023

    2023 arXiv

  40. [2022]

    Available: https://api.semanticscholar.org/CorpusID: 248476411

    [Online]. Available: https://api.semanticscholar.org/CorpusID: 248476411

  41. [2023]

    Available: https://api.semanticscholar.org/CorpusID: 258615266

    [Online]. Available: https://api.semanticscholar.org/CorpusID: 258615266

  42. [2024]

    Available: https://api.semanticscholar.org/CorpusID: 266844925

    [Online]. Available: https://api.semanticscholar.org/CorpusID: 266844925

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.