Pith. sign in

REVIEW 4 major objections 7 minor 68 references

Seeing Beyond the Scene: Enhancing Vision-Language Models with Interactional Reasoning

T0 review · 4 major / 7 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Training vision-language models on action-aware scene graphs—who does what to whom—improves interaction-heavy visual question answering by up to 7.9 percentage points over a standard LLaVA-v1.5 baseline.

desk verdict A coherent interaction-scene-graph pipeline with consistent benchmark gains, but the reward function at the core of the claimed contribution is never defined and the evidence for the RL stage is thin. read the letter →

arxiv 2505.09118 v1 pith:SQY7Y5BV submitted 2025-05-14 cs.CV

Dayong Liang , Changmeng Zheng , Zhiyuan Wen , Yi Cai , Xiao-Yong Wei , Qing Li This is my paper · ORCID
classification cs.CV
keywords sceneunderstandinginteractionalreasoningvision-languagemodelsgraphgenerationvisualquestionansweringreinforcementlearningchain-of-thoughtinteraction-augmented
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that vision-language models reason better about dynamic visual interactions—who is catching, throwing, blocking, or kicking—when they are trained on scene graphs that contain functional relationships in addition to spatial ones. It proposes ISGR, a framework that builds these interaction-augmented graphs through a summarize-and-align process and an interactional chain-of-thought, converts the graphs into training questions, and then reinforces the model with a reward signal for focus, disambiguation, and relevance. If the claim is right, a standard 7B model can gain roughly 7–8 percentage points on interaction-heavy benchmarks, match or surpass a much larger training set, and reach state-of-the-art multimodal question answering without changing its architecture.

What carries the argument

The load-bearing object is the interaction-augmented scene graph: a directed graph whose edges are subject–action–object triples grounded in bounding boxes, built by the Interactional Chain-of-Thought (ICoT) over an abstracted spatial graph. ICoT proceeds through subject identification, action recognition, object identification, and relation formalization, then applies saliency, grounding, and consistency constraints. The graph does triple duty: it is the supervision target for SFT, the raw material for four query types that generate object-object, subject-relation, relation-object, and comprehensive questions, and the input that the reward function scores with $R(y_k)=\lambda_1 F_{\text{focus}}(y_k)+\lambda_2 F_{\text{disamb}}(y_k)-\lambda_3 F_{\text{rele}}(y_k)$, with weights 0.4, 0.4, and 0.2. This triple role is what unifies graph construction and vision-language inference in a single optimization loop.

What would settle it

Regenerate the 300K interaction-augmented scene graphs with a different vision-language model generator, or shuffle the subject–action–object triples while keeping all other training data and rewards fixed; if the SEEDBench and MMT-Bench gains survive the shuffle, the graphs themselves are not the causal mechanism. A more direct version: build a small human-annotated interaction-graph test set and check whether models trained on ICoT graphs agree with humans more than a LLaVA-v1.5 baseline does.

Watch

Extended reading notes

Core claim

The central claim is that the bottleneck in scene reasoning is not spatial layout but functional interaction, and that this bottleneck can be removed through data and reward. The authors report that their full model, ISGR(SFT+IRR), outperforms the LLaVA-v1.5 baseline by +7.9% on SEEDBench, +7.8% on MMT-Bench, and +7.1% on RealWorldQA, and state that it reaches state-of-the-art results on multimodal question answering. The mechanism is a two-stage memory reinforcement: supervised fine-tuning on 1.3M samples that include 300K interaction-augmented scene graphs, followed by interaction reasoning reinforcement on just 500 curated interaction instructions using a reward composed of focus ($\lambda_1=0.4$), disambiguation ($\lambda_2=0.4$), and relevance ($\lambda_3=0.2$) terms. Because the same graphs serve as SFT targets, as the source of four query types, and as the grounding for rewards, construction and reasoning are optimized in one loop.

Load-bearing premise

The whole training signal rests on the assumption that the interaction graphs produced automatically by the ICoT process are accurate enough to teach real interaction reasoning, since the paper only says these graphs were 'manually verified' without saying how much or by whom.

Editorial extensions

If this is right

  • No architectural change is needed: a standard LLaVA-v1.5 7B model absorbs interaction reasoning through SFT plus a small reward-tuned stage, so the method can be applied to any instruction-tuned vision-language model.
  • Interaction-focused supervision is data-efficient: ISGR(SFT)-S, trained on 0.8M samples, stays competitive with the 1.2M LLaVA baseline, suggesting richer relational labels extract more learning per example.
  • Gains concentrate where interactions matter: on SEEDBench, Instance Interaction improves +7.3% and Instance Counting +31.4%, while Spatial Relation and Text Understanding decline slightly, showing the model trades some positional precision for functional understanding.
  • A small reward set goes far: 500 curated interaction instructions added through interaction reasoning reinforcement yield the largest jumps on RealWorldQA (+7.1), MMT-Bench (+7.8), and SEEDBench (+7.9) relative to the baseline.
  • Spatial grounding contributes consistently: removing bounding boxes from the graphs costs 0.5–2.2 points across benchmarks, though the reinforced model still beats the SFT-only model without boxes on most tasks.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial: If the ICoT graphs come from a strong external vision-language model, then ISGR is effectively a distillation of that model's interaction knowledge into an open 7B model; a quick way to test this is to compare ISGR(SFT) against a model trained on the same generator's captions without graph structure.
  • Editorial: The +31.4% gain on Instance Counting suggests that interaction role labels help the model individuate otherwise similar objects (e.g., players distinguished by role); this predicts improvements on dedicated counting and multi-object tracking benchmarks that the paper does not evaluate.
  • Editorial: The reward decomposition—focus, disambiguation, relevance—targets failure modes that appear in other structured reasoning outputs, so the same reward recipe could be tested on temporal event chains in video or on relation extraction from documents, not just static images.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 7 minor

Summary. The paper proposes Interaction-augmented Scene Graph Reasoning (ISGR), a framework aimed at improving vision-language models' reasoning about object interactions. The framework has three components: (i) a summarize-and-align graph constructor that builds spatial, abstract, and interaction graphs via an Interactional Chain-of-Thoughts (ICoT) process; (ii) generation of interaction-focused instruction-query pairs from these graphs for supervised fine-tuning; and (iii) Long-term Memory Reinforcement (LTMR) that combines SFT with a GRPO-based Interaction Reasoning Reinforcement (IRR) step using a composite reward. The authors evaluate ISGR variants on general VQA, spatial, and scene-reasoning benchmarks, reporting consistent gains over LLaVA-v1.5 and several baselines, with particularly large improvements on RealWorldQA, MMT-Bench, and SEEDBench. The paper includes ablations of query types, reward components, and spatial grounding, along with prompts, dataset statistics, and an algorithm pseudocode.

Significance. If the reported results hold, ISGR would be a useful contribution to structured interaction reasoning in VLMs, moving beyond spatial-only scene graphs and demonstrating that instruction tuning on interaction-augmented graphs can improve a range of multimodal benchmarks. The framework is modular, and the SFT data-construction pipeline is described in enough detail (prompts, dataset statistics, and ablation of query types) for others to reproduce the SFT component. However, the central IRR component is under-specified to the point of non-reproducibility, the closest scene-graph baselines are not compared, and the reported gains carry no uncertainty estimates. The paper therefore provides a promising but incompletely verified method; the SFT results and ablations are valuable, but the IRR claims need substantial additional specification and validation.

major comments (4)
  1. [Section 3.3.2, Eq. (15)] The reward function used in IRR is not defined. Eq. (15) gives R(y_k) = λ1·F_focus(y_k) + λ2·F_disamb(y_k) − λ3·F_rele(y_k), but the functions F_focus, F_disamb, and F_rele are never defined anywhere in the paper. Moreover, none of the three terms takes the ground-truth answer A_gt as input, even though Eq. (14) and the surrounding prose state that R evaluates both graph quality and answer correctness. As a result, L_IRR and the reported IRR gains (e.g., +7.9% on SEEDBench, +7.8% on MMT-Bench, +7.1% on RealWorldQA) cannot be reproduced or verified from the manuscript. Please provide exact operational definitions of F_focus, F_disamb, and F_rele (or cite a specific implementation), and clarify how the reward is used within the GRPO objective (e.g., advantage normalization, KL penalty, and number of sampled responses K).
  2. [Sections 2.3 and 4.3–4.4] The closest scene-graph-augmented baselines are missing from the experiments. CCoT [35], KM-COT [36], BDoG [54], and LLaVA-SG [43] are described in Related Work as the most related approaches, but none of them appears in Tables 1 or 2 or in the main-results discussion. Without these comparisons, the claim in Section 4.4 that ISGR 'reaches state-of-the-art results' on multimodal question answering is not supported; the observed gains over LLaVA-v1.5 could in principle be due to the larger SFT data mixture or the query-generation mechanism rather than the proposed interaction graph construction. Please add these baselines with matched training data and evaluation settings, or explicitly temper the SOTA claim.
  3. [Sections 3.2 and 3.3.1] The training signal for the interaction graphs has a potential circularity that is not addressed. The interaction-augmented graphs used as SFT targets and as part of the reward ground truth are generated by an unspecified multimodal LLM through the ICoT prompts in Section 3.2. Section 3.3.1 mentions 'manually verified interaction triplets' but does not specify which model produced the graphs, how many annotators were involved, what the agreement was, or how much of the 300K interaction data was manually verified. If the generator is from the same model family as the model being trained (or shares the same weights), the learned 'long-term memory' may simply reproduce generator biases rather than learn generalizable interaction reasoning. Please specify the generator model(s), the verification protocol, and any quality-control statistics.
  4. [Tables 1–3 and Section 4.4] All reported benchmark numbers appear to come from a single run or a single evaluation, with no error bars, multiple seeds, or statistical significance tests. This is especially problematic for Table 3, where the average performance differences between the full reward and each single-component ablation are at most 0.7 points (59.98 vs. 59.48 vs. 59.40 vs. 59.26) — well within typical run-to-run variance for 7B VLMs. The headline IRR gains in Table 1 and Figure 3 should be accompanied by variance estimates or, at minimum, results from multiple seeds, to establish that the IRR contribution is not noise.
minor comments (7)
  1. [Abstract] In the abstract, 'lone-term memory reinforcement' should be 'long-term memory reinforcement.'
  2. [Section 3.3.1] The text says 'derived from our ICoT aproach'; 'aproach' should be 'approach.'
  3. [Eq. (12)] The set notation in Eq. (12) has mismatched braces: D_interact = {(I_i, Q_i, Gfinal_i)}^N should be {(I_i, Q_i, Gfinal_i)}_{i=1}^N.
  4. [Table 4 and Section 4.1] The dataset size numbers are internally inconsistent: the small-scale variant is stated as 841K, but LLaVA-v1.5-mixed-665k (665K) plus 300K interaction-augmented data sums to 965K; the medium-scale variant is stated as 1,371K, but the listed components sum to 1,441K. Please clarify deduplication and the exact composition of each variant.
  5. [Algorithm 1] In Algorithm 1, the loop variable 'I' conflicts with the use of 'I' for the image; using a lowercase index (e.g., 'i') would avoid the confusion.
  6. [References] Reference [54] is incomplete, containing '[n. d.]' and missing the full proceedings information; also, the text uses 'KM-COT' while the reference list entry [36] is titled 'Kam-cot,' so the naming should be unified.
  7. [Figure 2] The figure contains informal elements such as 'IRR✨' and uses 'ICOT' in one label instead of the consistent 'ICoT'; please clean up the figure labels.

Circularity Check

1 steps flagged · score 6.0 of 10

Interaction-graph supervision is generated by the same model being trained, so the graph-construction signal is partly self-referential; benchmark evaluation remains independent.

  1. self definitional [Section 3.2 (Eq. 5), Section 3.3.1 (Eqs. 12–13), Algorithm 1]
    "The current scene graph G_i is updated by the multimodal LLM M with a set of operation functions F. ... G_t = f_ICoT(G_a,I,Q) (5). ... D_interact ={(I_i,Q_i,G_final_i)}^N_{i=1} (12). ... L_SFT = E_{(I,Q,G)∼D}[-log P_M(G|I,Q)] (13)."

    The SFT target G_final in Eq. (12) is defined as the output of the ICoT pipeline in Eq. (5), and that pipeline is executed using the same multimodal LLM M that Eq. (13) then fine-tunes (Algorithm 1 lists M as the model used for graph construction). The paper concedes the interactions come 'primarily generated through the ICoT rather than explicit human annotations', and the only independent anchor mentioned is a vague 'Manually verified interaction triplets' with no scale, protocol, or selection criteria. Thus the supervised signal for the central graph-construction claim is the model's own output; training on it can at best reinforce M's prior interaction tendencies rather than introduce externally verified interaction knowledge.

full rationale

The main empirical comparisons are against external benchmarks, so the reported headline gains are not tautological. However, the paper's central mechanism for acquiring 'interaction knowledge' is a closed loop: the interaction-augmented graphs used as SFT targets and as the basis for the claimed long-term memory are generated by the same MLLM family that is being trained, with only an unspecified amount of manual QA. This makes the graph-construction component partially self-referential. Separately, the IRR reward in Eq. (15) is underspecified and never references the ground-truth answer A_gt, which undermines the claimed correctness signal but is better characterized as a reproducibility gap than as a circular step. Because the benchmark evaluation is independent but the training-signal derivation is not, the overall circularity score is 6 rather than higher.

Assumptions & free parameters 2 free parameters · 5 assumptions · 0 invented entities

The central claim rests on the accuracy of self-generated interaction graphs and on an unspecified reward mechanism. The paper contributes the pipeline design but does not provide independent evidence for graph quality or reward computation, so the reader must accept these as assumptions.

free parameters (2)
  • Reward weights lambda1, lambda2, lambda3 = 0.4, 0.4, 0.2
    Selected via 'extensive experimentation' on the evaluation benchmarks (Section 3.3.2); no held-out validation set is described.
  • IRR instruction set size = 500
    A curated set of 500 high-quality interaction instructions (Section 4.1); the curation criteria are not specified, making this a design choice that affects the reward training.
assumptions (5)
  • domain assumption The ICoT process produces interaction graphs accurate enough to serve as training targets and reward ground truth.
    Section 3.2 defines ICoT as prompted inference; Section 3.3.1 uses these graphs to build D_interact. Only a vague 'manually verified' statement supports their correctness.
  • domain assumption The reward components F_focus, F_disamb, and F_rele can be computed and align with reasoning quality.
    Eq. 15 in Section 3.3.2 introduces these functions but gives no implementation or definition, so the paper relies on an unstated assumption that such functions exist and are measurable.
  • domain assumption Targeted interaction queries activate latent functional knowledge in the base VLM.
    Section 3.2.3 and Section 1 state this premise; no experiment isolates or validates this activation mechanism.
  • domain assumption GRPO-style policy optimization with the proposed reward improves generalization to unseen interactions.
    Section 3.3.2 invokes GRPO without citation or derivation; the effectiveness is an empirical assumption tested only indirectly on benchmarks.
  • domain assumption The evaluation benchmarks (SEEDBench, MMT-Bench, RealWorldQA, etc.) measure interactional reasoning as intended.
    Section 4.2 selects benchmarks by category but provides no analysis that the gains reflect interaction reasoning specifically rather than other confounds.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Seeing Beyond the Scene: Enhancing Vision-Language Models with Interactional Reasoning." pith.science (2026). https://pith.science/paper/SQY7Y5BV

@misc{pith2026250509118,
  author       = {Pith},
  title        = {Pith review of: Seeing Beyond the Scene: Enhancing Vision-Language Models with Interactional Reasoning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/SQY7Y5BV}},
  note         = {Machine review of arXiv:2505.09118}
}
read the original abstract

Traditional scene graphs primarily focus on spatial relationships, limiting vision-language models' (VLMs) ability to reason about complex interactions in visual scenes. This paper addresses two key challenges: (1) conventional detection-to-construction methods produce unfocused, contextually irrelevant relationship sets, and (2) existing approaches fail to form persistent memories for generalizing interaction reasoning to new scenes. We propose Interaction-augmented Scene Graph Reasoning (ISGR), a framework that enhances VLMs' interactional reasoning through three complementary components. First, our dual-stream graph constructor combines SAM-powered spatial relation extraction with interaction-aware captioning to generate functionally salient scene graphs with spatial grounding. Second, we employ targeted interaction queries to activate VLMs' latent knowledge of object functionalities, converting passive recognition into active reasoning about how objects work together. Finally, we introduce a lone-term memory reinforcement learning strategy with a specialized interaction-focused reward function that transforms transient patterns into long-term reasoning heuristics. Extensive experiments demonstrate that our approach significantly outperforms baseline methods on interaction-heavy reasoning benchmarks, with particularly strong improvements on complex scene understanding tasks. The source code can be accessed at https://github.com/open_upon_acceptance.

Figures

Figures reproduced from arXiv: 2505.09118 by the authors.

Figure 1
Figure 1. Examples showing how our interaction-augmented scene graphs enhance reasoning on dynamic interactions. Spatial: [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Overview of our Interaction-augmented Scene Graph Reasoning (ISGR) framework: (a) Summarize-and-Align Graph [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Performance comparison on scene reasoning bench [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Category Performance Comparison on SEEDBench. [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: A case study for our proposed ISGR framework: (Left) Limitations of spatial reasoning where models provide [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

68 extracted references · 27 canonical work pages

  1. [35]

    Hengyue Liu, Ning Yan, Masood Mortazavi, and Bir Bhanu. 2021. Fully convo- lutional scene graph generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . 11546–11556

    work page 2021

  2. [36]

    Debjyoti Mondal, Suraj Modi, Subhadarshi Panda, Rituraj Singh, and Go- dawari Sudhakar Rao. 2024. Kam-cot: Knowledge augmented multimodal chain- of-thoughts reasoning. In Proceedings of the AAAI conference on artificial intelli- gence, Vol. 38. 18798–18806

    work page 2024

  3. [54]

    Changmeng Zheng, DaYong Liang, Wengyu Zhang, Xiaoyong Wei, Tat-Seng Chua, and Qing Li. [n. d.]. A Picture Is Worth a Graph: A Blueprint Debate Paradigm for Multimodal Reasoning. In ACM Multimedia 2024

    work page 2024

  4. [43]

    Sebastian Schuster, Ranjay Krishna, Angel Chang, Li Fei-Fei, and Christopher D Manning. 2015. Generating semantically precise scene graphs from textual descriptions for improved image retrieval. In Proceedings of the fourth workshop MM ’25, October 27–31, 2025, Dublin, Ireland Dayong Liang, Changmeng Zheng, Zhiyuan Wen, Yi Cai, Xiao-Yong Wei, and Qing Li ...

    work page 2015

  5. [1]

    Anas Awadalla, Irena Gao, Josh Gardner, Jack Hessel, Yusuf Hanafy, Wanrong Zhu, Kalyani Marathe, Yonatan Bitton, Samir Gadre, Shiori Sagawa, et al. 2023. Openflamingo: An open-source framework for training large autoregressive vision-language models. arXiv preprint arXiv:2308.01390 (2023)

    arXiv 2023

  6. [2]

    Xiaojun Chang, Pengzhen Ren, Pengfei Xu, Zhihui Li, Xiaojiang Chen, and Alex Hauptmann. 2021. A comprehensive survey of scene graphs: Generation and application. IEEE Transactions on Pattern Analysis and Machine Intelligence 45, 1 (2021), 1–26

    work page 2021

  7. [3]

    Guikun Chen, Jin Li, and Wenguan Wang. 2024. Scene Graph Generation with Role-Playing Large Language Models. In NeurIPS

    work page 2024

  8. [4]

    Guiming Hardy Chen, Shunian Chen, Ruifei Zhang, Junying Chen, Xiangbo Wu, Zhiyi Zhang, Zhihong Chen, Jianquan Li, Xiang Wan, and Benyou Wang. 2024. Allava: Harnessing gpt4v-synthesized data for lite vision-language models. arXiv preprint arXiv:2402.11684 (2024)

    arXiv 2024

Show all 68 references
  1. [5]

    Keqin Chen, Zhao Zhang, Weili Zeng, Richong Zhang, Feng Zhu, and Rui Zhao

  2. [6]

    Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Conghui He, Jiaqi Wang, Feng Zhao, and Dahua Lin. 2024. Sharegpt4v: Improving large multi-modal models with better captions. In European Conference on Computer Vision . Springer, 370– 387

    2024

  3. [7]

    Yuren Cong, Michael Ying Yang, and Bodo Rosenhahn. 2023. Reltr: Relation transformer for scene graph generation. IEEE Transactions on Pattern Analysis and Machine Intelligence 45, 9 (2023), 11169–11183

    2023

  4. [8]

    Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. 2023. InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning. arXiv:2305.06500 [cs.CV] https://arxiv.org/abs/2305.06500

    2023 arXiv

  5. [9]

    William Fedus, Barret Zoph, and Noam Shazeer. 2022. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research 23, 120 (2022), 1–39

    2022

  6. [10]

    Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh

  7. [11]

    Qiao Gu, Ali Kuwajerwala, Sacha Morin, Krishna Murthy Jatavallabhula, Bipasha Sen, Aditya Agarwal, Corban Rivera, William Paul, Kirsty Ellis, Rama Chellappa, et al. 2024. Conceptgraphs: Open-vocabulary 3d scene graphs for perception and planning. In 2024 IEEE International Con...

    2024

  8. [12]

    Danna Gurari, Qing Li, Abigale J Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jiebo Luo, and Jeffrey P Bigham. 2018. Vizwiz grand challenge: Answering visual questions from blind people. In Proceedings of the IEEE conference on computer vision and pattern recognition . 3608–3617

    2018

  9. [13]

    Tao He, Lianli Gao, Jingkuan Song, and Yuan-Fang Li. 2022. Towards open- vocabulary scene graph generation with prompt-based finetuning. In European Conference on Computer Vision . Springer, 56–73

    2022

  10. [14]

    Yilun Hua and Yoav Artzi. 2024. Talk Less, Interact Better: Evaluating In-context Conversational Adaptation in Multimodal LLMs. arXiv preprint arXiv:2408.01417 (2024)

    2024 arXiv

  11. [15]

    Yufeng Huang, Jiji Tang, Zhuo Chen, Rongsheng Zhang, Xinfeng Zhang, Weijie Chen, Zeng Zhao, Zhou Zhao, Tangjie Lv, Zhipeng Hu, et al. 2024. Structure-clip: Towards scene graph knowledge to enhance multi-modal structured represen- tations. In Proceedings of the AAAI conference ...

    2024

  12. [16]

    Drew A Hudson and Christopher D Manning. 2019. Gqa: A new dataset for real- world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition . 6700–6709

    2019

  13. [17]

    Ruili Jiang, Kehai Chen, Xuefeng Bai, Zhixuan He, Juntao Li, Muyun Yang, Tiejun Zhao, Liqiang Nie, and Min Zhang. 2024. A survey on human preference learning for large language models. arXiv preprint arXiv:2406.11191 (2024)

    2024 arXiv

  14. [18]

    Justin Johnson, Ranjay Krishna, Michael Stark, Li-Jia Li, David Shamma, Michael Bernstein, and Li Fei-Fei. 2015. Image retrieval using scene graphs. InProceedings of the IEEE conference on computer vision and pattern recognition . 3668–3678

    2015

  15. [19]

    Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al

  16. [20]

    Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper Uijlings, Ivan Krasin, Jordi Pont-Tuset, Shahab Kamali, Stefan Popov, Matteo Malloci, Alexander Kolesnikov, et al. 2020. The open images dataset v4: Unified image classification, object detection, and visual relationship detec...

    2020

  17. [21]

    Junlin Lee, Yequan Wang, Jing Li, and Min Zhang. 2024. Multimodal Reasoning with Multimodal Knowledge Graph. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) . 10767– 10782

    2024

  18. [22]

    International journal of computer vision 123 (2017), 32–73

    Visual genome: Connecting language and vision using crowdsourced dense image annotations. International journal of computer vision 123 (2017), 32–73

    2017

  19. [23]

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2023. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning . PMLR, 19730–19742

    2023

  20. [24]

    Rongjie Li, Songyang Zhang, and Xuming He. 2022. Sgtr: End-to-end scene graph generation with transformer. In proceedings of the IEEE/CVF conference on computer vision and pattern recognition . 19486–19496

    2022

  21. [25]

    Bohao Li, Yuying Ge, Yixiao Ge, Guangzhi Wang, Rui Wang, Ruimao Zhang, and Ying Shan. 2024. Seed-bench: Benchmarking multimodal large language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 13299–13308

    2024

  22. [26]

    Xin Lin, Changxing Ding, Jinquan Zeng, and Dacheng Tao. 2020. Gps-net: Graph property sensing network for scene graph generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . 3746–3753

    2020

  23. [27]

    Xin Lin, Changxing Ding, Jing Zhang, Yibing Zhan, and Dacheng Tao. 2022. Ru-net: Regularized unrolling network for scene graph generation. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . 19457– 19466

    2022

  24. [28]

    Bin Lin, Zhenyu Tang, Yang Ye, Jiaxi Cui, Bin Zhu, Peng Jin, Jinfa Huang, Junwu Zhang, Yatian Pang, Munan Ning, et al. 2024. Moe-llava: Mixture of experts for large vision-language models. arXiv preprint arXiv:2401.15947 (2024)

    2024 arXiv

  25. [29]

    Fuxiao Liu, Kevin Lin, Linjie Li, Jianfeng Wang, Yaser Yacoob, and Lijuan Wang

  26. [30]

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023. Visual in- struction tuning. Advances in neural information processing systems 36 (2023), 34892–34916

    2023

  27. [31]

    Fangyu Liu, Guy Emerson, and Nigel Collier. 2023. Visual spatial reasoning. Transactions of the Association for Computational Linguistics 11 (2023), 635–651

    2023

  28. [32]

    Wei Liu, Weihao Zeng, Keqing He, Yong Jiang, and Junxian He. 2023. What makes good data for alignment? a comprehensive study of automatic data selection in instruction tuning. arXiv preprint arXiv:2312.15685 (2023)

    2023 arXiv

  29. [33]

    arXiv preprint arXiv:2306.14565 (2023)

    Mitigating hallucination in large multi-modal models via robust instruction tuning. arXiv preprint arXiv:2306.14565 (2023)

    2023 arXiv

  30. [34]

    Keming Lu, Hongyi Yuan, Zheng Yuan, Runji Lin, Junyang Lin, Chuanqi Tan, Chang Zhou, and Jingren Zhou. 2023. # instag: Instruction tagging for analyzing supervised fine-tuning of large language models. arXiv preprint arXiv:2308.07074 (2023)

    2023 arXiv

  31. [37]

    Shayne Longpre, Le Hou, Tu Vu, Albert Webson, Hyung Won Chung, Yi Tay, Denny Zhou, Quoc V Le, Barret Zoph, Jason Wei, et al. 2023. The flan collection: Designing data and methods for effective instruction tuning. In International Conference on Machine Learning . PMLR, 22631–22648

    2023

  32. [38]

    Jack W Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, et al. 2021. Scaling language models: Methods, analysis & insights from training gopher. arXiv preprint arXiv:2112.11446 (2021)

    2021 arXiv

  33. [39]

    Chancharik Mitra, Brandon Huang, Trevor Darrell, and Roei Herzig. 2024. Compo- sitional chain-of-thought prompting for large multimodal models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . 14420– 14431

    2024

  34. [40]

    Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. 2019. Towards vqa models that can read. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 8317–8326

    2019

  35. [41]

    Baolin Peng, Chunyuan Li, Pengcheng He, Michel Galley, and Jianfeng Gao. 2023. Instruction tuning with gpt-4. arXiv preprint arXiv:2304.03277 (2023)

    2023 arXiv

  36. [42]

    Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kul- shreshtha, Heng-Tze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, et al. 2022. Lamda: Language models for dialog applications. arXiv preprint arXiv:2201.08239 (2022)

    2022 arXiv

  37. [44]

    Junke Wang, Lingchen Meng, Zejia Weng, Bo He, Zuxuan Wu, and Yu-Gang Jiang. 2023. To see is to believe: Prompting gpt-4v for better visual instruction tuning. arXiv preprint arXiv:2311.07574 (2023)

    2023 arXiv

  38. [45]

    Quan Sun, Qiying Yu, Yufeng Cui, Fan Zhang, Xiaosong Zhang, Yueze Wang, Hongcheng Gao, Jingjing Liu, Tiejun Huang, and Xinlong Wang. 2023. Emu: Generative pretraining in multimodality. arXiv preprint arXiv:2307.05222 (2023)

    2023 arXiv

  39. [46]

    Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishirzi. 2023. Self-Instruct: Aligning Language Mod- els with Self-Generated Instructions. In Proceedings of the 61st Annual Meeting of the Association for Computational Ling...

    2023

  40. [47]

    Jingyi Wang, Jianzhong Ju, Jian Luan, and Zhidong Deng. 2025. LLaVA-SG: Leveraging scene graphs as visual semantic expression in vision-language models. In ICASSP 2025-2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 1–5

    2025

  41. [48]

    Zhiyang Xu, Chao Feng, Rulin Shao, Trevor Ashby, Ying Shen, Di Jin, Yu Cheng, Qifan Wang, and Lifu Huang. 2024. Vision-flan: Scaling human-labeled tasks in visual instruction tuning. arXiv preprint arXiv:2402.11690 (2024)

    2024 arXiv

  42. [49]

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin. 2024. Qwen2-VL: Enhancing Vision-Language Mode...

    2024 arXiv

  43. [50]

    Qifan Yu, Juncheng Li, Yu Wu, Siliang Tang, Wei Ji, and Yueting Zhuang. 2023. Visually-prompted language model for fine-grained scene graph generation in an open world. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 21560–21571

    2023

  44. [51]

    x.ai. 2024. Grok-1.5 Vision Preview. https://x.ai/news/grok-1.5v Accessed: 2025-03-23

    2024

  45. [52]

    Yong Zhang, Yingwei Pan, Ting Yao, Rui Huang, Tao Mei, and Chang-Wen Chen

  46. [53]

    Kaining Ying, Fanqing Meng, Jin Wang, Zhiqian Li, Han Lin, Yue Yang, Hao Zhang, Wenbo Zhang, Yuqi Lin, Shuo Liu, et al. 2024. MMT-Bench: A Comprehensive Multimodal Benchmark for Evaluating Large Vision-Language Models Towards Multitask AGI. In International Conference on Machi...

    2024

  47. [55]

    Hang Zhang, Xin Li, and Lidong Bing. 2023. Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understanding. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: System Demonstrations. 543–553

    2023

  48. [56]

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems 36 (2023), 46595–46623

    2023

  49. [57]

    In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Learning to generate language-supervised and open-vocabulary scene graph using pre-trained visual-semantic space. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . 2915–2924

  50. [58]

    Zicheng Zhang, Haoning Wu, Chunyi Li, Yingjie Zhou, Wei Sun, Xiongkuo Min, Zijian Chen, Xiaohong Liu, Weisi Lin, and Guangtao Zhai. 2024. A-bench: Are lmms masters at evaluating ai-generated images? arXiv preprint arXiv:2406.03070 (2024)

    2024 arXiv

  51. [59]

    Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. 2023. Minigpt-4: Enhancing vision-language understanding with advanced large lan- guage models. arXiv preprint arXiv:2304.10592 (2023). A More Experimental Details A.1 Algorithm for ISGR For a better understa...

    2023 arXiv

  52. [60]

    Chaofan Zheng, Xinyu Lyu, Lianli Gao, Bo Dai, and Jingkuan Song. 2023. Prototype-based embedding network for scene graph generation. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . 22783–22792

    2023

  53. [62]

    Chunting Zhou, Pengfei Liu, Puxin Xu, Srinivasan Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, et al . 2023. Lima: Less is more for alignment. Advances in Neural Information Processing Systems 36 (2023), 55006– 55021

    2023

  54. [63]

    Zijian Zhou, Miaojing Shi, and Holger Caesar. 2023. VLPrompt: Vision-Language Prompting for Panoptic Scene Graph Generation. arXiv preprint arXiv:2311.16492 (2023)

    2023 arXiv

  55. [65]

    Q: What is the relationship between object1[bbox] and object2[bbox]? A: object1 relation object2

  56. [66]

    Q: What does object1[bbox] relation? A: object1 relation object2[bbox]

  57. [67]

    Q: What is relation by object2[bbox]? A: object1[bbox] relation object2

  58. [68]

    Q: What objects have a relationship with object1[bbox]? A: object1 relation1 object2[bbox], relation2 object3[bbox], etc. When creating questions: - Focus on the main subject as provided in the scene graph MM ’25, October 27–31, 2025, Dublin, Ireland Dayong Liang, Changmeng Zh...

    2025

  59. [2017]

    In Proceedings of the IEEE conference on computer vision and pattern recognition

    Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition . 6904–6913

  60. [2023]

    arXiv preprint arXiv:2306.15195 (2023)

    Shikra: Unleashing multimodal llm’s referential dialogue magic. arXiv preprint arXiv:2306.15195 (2023)

    2023 arXiv

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.