Pith. sign in

REVIEW 4 major objections 6 minor 2 cited by

Open World Scene Graph Generation using Vision Language Models

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Pretrained VLMs can generate scene graphs over unseen objects and relations without any training, and this paper supplies a standard protocol for measuring that open-world ability.

desk verdict Useful first benchmark for fully open-world SGG with a training-free VLM pipeline, but the empirical claims need tightening before the numbers are taken at face value. read the letter →

arxiv 2506.08189 v1 pith:43MKWYDH submitted 2025-06-09 cs.CV cs.CL

Amartya Dutta , Kazi Sajeed Mehrab , Medha Sawhney , Abhilash Neog , Mridul Khurana , Sepideh Fatemi , Aanish Pradhan , M. Maruf
show 3 more authors
Ismini Lourentzou Arka Daw Anuj Karpatne
This is my paper
classification cs.CVcs.CL
keywords scenegraphgenerationopen-worldzero-shotvision-languagemodelstraining-freepairrefinementopenvocabulary
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether an off-the-shelf vision-language model can produce a scene graph—a structured list of objects in an image and the relationships between them—when the object categories and relationship types were never seen during any task-specific training. The authors claim yes: their training-free pipeline, called OwSGG, extracts candidate objects by prompting a VLM, maps those names onto dataset vocabularies with a contrastive text encoder, localizes them with an open-vocabulary detector, prunes implausible pairs using a mix of semantic and depth-based scores, and then asks the VLM to name the relationship for each surviving pair. They also formalize an Open-World evaluation protocol that splits the test vocabulary into closed, zero-shot, open-vocabulary, and fully open-world settings, and they report non-zero recall on Visual Genome, Open Images V6, and PSG. A sympathetic reader would care because success would mean relational visual understanding is already latent in pretrained multimodal models, so scene-graph generation could be performed without dataset-specific fine-tuning, and the new protocol would let future systems be compared on that capability.

What carries the argument

The load-bearing mechanism is the pair-refinement module, which fuses a VLM semantic score $P^S_{ij} = \mathrm{VLM}(o_i, o_j)$ with a geometric score $P^G_{ij} = \sigma(-\beta(d_{ij}-\tau))$ computed from 2D bounding-box centers and monocular depth estimates, then keeps the top-$k$ pairs under the weighted log combination $P^{\text{combined}}_{ij} = \alpha \log P^S_{ij} + (1-\alpha)\log P^G_{ij}$. This is what turns the combinatorially large set of possible object pairs into a short, VLM-processable candidate list before the relation-prediction step; the semantic branch is what supplies open-world generalization, since it judges whether two category names plausibly interact without needing to have seen that exact pair, while the depth branch supplies spatial plausibility.

What would settle it

Take an image whose ground-truth scene graph contains a pair that is visually interacting but semantically unusual (for example, a squirrel sitting on a drone, or a person holding an octopus): the pair-refinement module must rank that pair in its top-$k$ using only the names 'squirrel/drone' or 'person/octopus', and the final recalled triplet must appear in the output. If such visually obvious but text-unusual pairs are consistently pruned, the claim that VLMs provide open-world relational understanding would be shown to be largely a spatial-semantic prior rather than a true image-conditioned reasoning ability.

Watch

Extended reading notes

Core claim

The central discovery is that pretrained VLMs, guided by a fixed prompting and filtering pipeline and never fine-tuned on scene-graph data, can recover ground-truth relational triplets even when both the object names and the predicate names are novel relative to any training set; the paper's Table 3 reports R@50/R@100 of 1.92/2.56 for LLaVA-next and 1.61/2.41 for Qwen2-72B in the strictly Open-World setting on Visual Genome. The paper reads this as evidence that the semantic priors in VLMs are sufficient for compositional relational reasoning, not just for recognizing isolated objects. The framework is claimed to be model-agnostic in that the same pipeline was run with three different VLM backbones and produced the same qualitative pattern of results.

Load-bearing premise

The pipeline's ceiling is set by a text-only judgment: the VLM is asked, without seeing the image, how likely two object category names are to interact, and any pair that gets a low score is discarded before the image is ever consulted, so visually present but semantically unusual interactions are lost before the final relation prediction step.

Editorial extensions

If this is right

  • If the claim holds, scene graph generation becomes a deployment-time capability of any sufficiently capable VLM plus a detector and a text encoder; no SGG-specific training runs are needed to add a new domain's objects or predicates.
  • The new Open-World split (novel objects and novel relations simultaneously) gives later methods a concrete, standardized baseline to beat, which the paper argues was previously missing.
  • Because the pipeline is model-agnostic, the same prompts and refinement code can be re-run as stronger VLMs appear, making performance on open-world SGG a direct comparison of the VLM's latent relational knowledge rather than of trained SGG architectures.
  • The paper's results in OVR on PSG (R@50/100 of 10.42/13.54 for Qwen2-72B, above the PGSG baseline) suggest that on some open-world axes a training-free VLM pipeline can match or exceed trained open-vocabulary methods, which would redirect effort toward better prompting and pair selection rather than dataset-specific supervision.
  • The qualitative ablation results (depth-only refinement winning in closed vocabulary, semantic-plus-depth winning in open settings) imply that as the evaluation domain becomes less familiar, the VLM's name-based semantic prior contributes more than spatial proximity.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable extension the paper leaves implicit: the semantic compatibility score $\mathrm{VLM}(o_i, o_j)$ could be replaced by a contrastive text-encoder score (e.g., the same SimCSE embeddings used in entity mapping), and comparing those two variants would isolate whether the VLM's generative semantic judgment adds anything beyond embedding similarity for pair recall.
  • A second extension: because the paper's pipeline is fully training-free, the same protocol could be run with the VLM replaced by a much larger or vision-centric model, or with the detector replaced by a different open-vocabulary detector, to measure how much of the open-world recall ceiling is set by each module rather than by the VLM itself.
  • The paper reports non-zero OW recall but does not analyze where the recalled triplets come from; a reasonable inference is that they are dominated by spatial relations (e.g., 'on', 'next to', 'behind') whose names co-occur with the novel object names in the VLM's training corpus, so the headline open-world numbers may be measuring commonsense spatial priors rather than truly novel functional relat
  • The authors' choice to evaluate only on test data with no training could be re-purposed as a data-annotation tool: a deployed OwSGG pipeline could pre-label candidate triplets for new object vocabularies, with a human only correcting the filtered pairs, which would lower the cost of building future SGG datasets.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes OwSGG, a training-free pipeline for open-world scene graph generation that uses pretrained vision-language models (VLMs) as the sole relational reasoner. The pipeline consists of VLM-based entity generation, SimCSE-based entity mapping to dataset categories, Grounding DINO detection, a pair-refinement stage that fuses an image-agnostic semantic compatibility score with a geometric distance score, and a final VLM prompt that predicts relations for the retained top-k pairs. The authors define a taxonomy of open-world settings (Close Vocabulary, Zero-Shot, OVR, OVD, OVD+R, and strictly Open World) and report recall/mRecall on VG150, OIV6, and PSG for LLaVA-Next-7B, Qwen2-VL-7B, and Qwen2-VL-72B. The main empirical claim is that VLMs produce non-zero open-world recall without any task-specific training, with the strongest OW numbers on VG150 (e.g., Qwen2-VL-72B R@50=1.61, R@100=2.41 in Table 3).

Significance. If the central claim holds, the paper makes a useful contribution: it provides a reproducible-style evaluation protocol for training-free open-world SGG, a lightweight pair-refinement strategy, and evidence that large VLMs contain enough relational knowledge to generate non-trivial scene graphs over novel object and relation vocabularies. The paper tests three VLM backbones on three standard datasets and includes ablations of the semantic and geometric refinement branches, which is a strength. The reported non-zero recall in the strictly open-world setting is a genuinely interesting finding. However, the attribution of that recall to visual understanding, rather than to language-only priors or geometry, is not yet isolated, and the evaluation protocol has several gaps concerning hyperparameter selection, output parsing, and baseline generation. These issues are fixable but must be addressed before the central claim is fully supported.

major comments (4)
  1. [§3.3.4, Eq. (1), and Table 3] The paper does not include an image-blind control, so the non-zero OW recall in Table 3 cannot be attributed to visual understanding. The semantic pair-refinement score in Eq. (1), P^S_ij = VLM(o_i, o_j), is computed from category labels alone and assigned to every instance of the pair, then fused in Eq. (4) to select the top-k pairs before any image-conditioned relation prediction. Because VLM pretraining encodes strong language priors about which object categories typically interact, a correct OW triplet could in principle be produced from object names and box geometry without the model ever seeing pixel content. The authors should run a control that removes or degrades the image input (e.g., blank image, no image, or a text-only variant of the relation-generation step) and report whether the OW recall persists. Without such a control, the 'vision-language' interpretation of the central claim is not isolated.
  2. [§4.4 and Supplementary B.5] The hyperparameters appear to be selected on the same data used for reporting, and no error bars or validation splits are provided. The final values (alpha=0.25, beta=10 for 72B and beta=16 for 7B models, top_k=25, tau=0.2, delta=0.05, k=2, lambda1=1.0, lambda2=1.5, distance threshold 0.5) are fixed in the supplementary, while Fig. 2 ablates alpha and beta with no mention of a held-out split. Since the reported R@K numbers are the main evidence for the central claim, the authors should either commit to a validation-based selection procedure or provide a sensitivity analysis showing that the OW results are robust across a range of hyperparameters. The lack of error bars is also a concern for a method that involves stochastic VLM decoding, even at temperature 0.1.
  3. [Table 3 and §4.3] The only non-zero OW baseline in Table 3 is OvSGTR with a dagger indicating 'results were generated for this work,' but the paper does not describe how these numbers were produced, what split filtering was applied, or whether the same OW triplet definition was used. Without this protocol description, the comparison cannot be verified or reproduced. Additionally, the corresponding table row may have been generated with different code than the original OvSGTR release, so the authors should either provide the exact protocol and code or clearly state that these numbers are preliminary. This is load-bearing because Table 3 is the only direct evidence that the proposed open-world setting is distinguishable from the closed-vocabulary baselines.
  4. [§3.3.5 and §4 (Evaluation Metrics)] The paper never specifies how free-form VLM relations (e.g., 'the woman is sitting on the chair') are mapped to the fixed predicate vocabularies of VG150, PSG, and OIV6, nor how multiple generated sentences per pair are converted into the confidence scores that underlie R@K and mR@K. This mapping is essential for reproducing any of the reported recall numbers and for understanding whether the evaluation is lenient (e.g., accepting synonym paraphrases) or strict (exact string match). The authors should describe the parsing, normalization, and matching procedure, and ideally release the mapping code. Without this, the numerical results in Tables 1-4 are not independently checkable.
minor comments (6)
  1. [Table 2] The PredCls OVR rows for OwSGG on VG contain three comma-separated values per cell (e.g., '0.75 / 1.36 / 1.5' and '0.44 / 1.2 / 2.12'), while the table header specifies only mR@50/100 and R@50/100. This is likely a transcription error and should be corrected to two values per column.
  2. [Table 1] The row label 'HEIRCOM' in the OIV6 PredCls block should be 'VCTree+HIERCOM' for consistency with the VG block; also, the CaCao row in the VG PredCls block reports only three values under Close Vocabulary, leaving the reader unsure whether mR@20 is missing or intentionally blank.
  3. [Figure 2] The caption says 'F1 scores across different (a) α and (b) β values for the Qwen-72B model,' but the figure legends include llava-next, qwen2_vl, and qwen2_72vl. The caption should be corrected to describe all three models.
  4. [Section 2 and Supplementary B.3.1] There is a typo 'V ocabualry' in the section title, and the entity mapping ablation text says 'SBERT slightly outperforms SimCSE for Qwen72 on PSG' while the figure caption claims 'SimCSE yields up to a 5% recall boost on PSG and OI'; these statements are not obviously consistent and should be reconciled.
  5. [References] References [47] and [48] are duplicates of the same paper; one should be removed and all citations updated.
  6. [General] No code or public evaluation scripts are released, which is particularly important for a paper whose main deliverable is a benchmark protocol and a set of baselines; releasing the parser, split definitions, and hyperparameter configuration would substantially strengthen reproducibility.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the reported scene-graph results are generated by an image-conditioned VLM on external benchmarks, not by a fitted equation or self-citation chain.

full rationale

The paper's central claim is that a pretrained VLM, guided by the proposed pipeline, can generate scene graphs over unseen object and relation vocabularies with zero additional learning. That claim is evaluated on external benchmarks (VG150, OIV6, PSG) using standard Recall@K metrics, and the final relation labels are produced by prompting a pretrained VLM with the image and candidate object pairs (Section 3.3.5). Relation predictions therefore come from the VLM's image-conditioned outputs rather than from an equation that is equivalent to its inputs. The text-only semantic pair score P^S_ij = VLM(o_i, o_j) (Eq. 1) and the geometric score P^G_ij (Eq. 3) are combined in Eq. 4 to select candidate pairs; this does prune the search space, but it does not by construction force the reported recall values, since the VLM still has to produce the correct relation for a retained pair. No load-bearing self-citations appear: the references are to external prior work, and no argument in the paper reduces to a citation by the same authors. The main legitimate weakness is that hyperparameters such as alpha=0.25 and beta=10/16 are chosen with the aid of ablations (Section 4.4 and Supplementary B.5.2) on the same test splits used for reporting, since the method has no training split. This is a test-set-peeking concern that can inflate absolute numbers, but it does not make the derivation circular: no reported prediction is equal by construction to a fitted parameter or an assumed ground-truth label. The paper also openly acknowledges component-error and context-length limitations. Overall, the derivation chain is self-contained and empirically anchored to external data, so the circularity score is 0.

Assumptions & free parameters 7 free parameters · 5 assumptions · 0 invented entities

The framework has no new theoretical entities, but it introduces several tunable hyperparameters and borrows modules from prior work, which shifts the burden of performance onto component quality and parameter selection.

free parameters (7)
  • alpha (fusion weight) = 0.25
    Chosen via ablation in Fig 2a; no separate validation split is reported, so it may be fitted to the evaluation data.
  • beta (sigmoid sharpness) = 16 for 7B, 10 for 72B
    Tuned per model in Fig 2b; affects which pairs are kept in geometric refinement.
  • top_k (retained candidate pairs) = 25 (in ablations; main experiments do not state it)
    Controls the number of pairs sent to the VLM relation generator; recall scales with it.
  • delta (near-max threshold for entity mapping) = 0.05
    Threshold for retaining close semantic matches; chosen by the authors.
  • k (max retained categories per entity) = 2
    Limits mapping candidates, affecting entity mapping precision and recall.
  • tau (softmax temperature for entity mapping) = 0.2
    Sharpens the similarity distribution; chosen by the authors.
  • geometric weights lambda1, lambda2, and distance threshold tau = 1.0, 1.5, 0.5
    Taken from prior work [6], but still free parameters that influence which pairs survive geometric filtering.
assumptions (5)
  • domain assumption Grounding DINO can localize novel object categories not seen in SGG training data.
    Entity detection relies on Grounding DINO's open-set detection (Section 3.3.3); if it misses novel objects, SgDet recall is capped.
  • domain assumption The VLM's semantic pair score, conditioned only on category names, is a meaningful proxy for visual interaction likelihood.
    Eq. 1 assigns compatibility scores without image evidence; the whole pair-refinement design assumes this text-only signal is useful.
  • domain assumption Pretrained VLMs contain enough relational knowledge to answer the relation prompts for unseen categories.
    The final scene graph generation step relies entirely on VLM outputs (Section 3.3.5).
  • domain assumption The standard SGG datasets and their ground-truth labels are valid proxies for evaluating general scene graph quality.
    All conclusions are based on recall against these annotations.
  • domain assumption Splits and novelty definitions from prior works (T-CAR, OvSGTR, Pix2Grp) correctly instantiate the paper's open-world taxonomy.
    The evaluation protocol depends on these external scripts, as described in Appendix A.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Open World Scene Graph Generation using Vision Language Models." pith.science (2026). https://pith.science/paper/43MKWYDH

@misc{pith2026250608189,
  author       = {Pith},
  title        = {Pith review of: Open World Scene Graph Generation using Vision Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/43MKWYDH}},
  note         = {Machine review of arXiv:2506.08189}
}
read the original abstract

Scene-Graph Generation (SGG) seeks to recognize objects in an image and distill their salient pairwise relationships. Most methods depend on dataset-specific supervision to learn the variety of interactions, restricting their usefulness in open-world settings, involving novel objects and/or relations. Even methods that leverage large Vision Language Models (VLMs) typically require benchmark-specific fine-tuning. We introduce Open-World SGG, a training-free, efficient, model-agnostic framework that taps directly into the pretrained knowledge of VLMs to produce scene graphs with zero additional learning. Casting SGG as a zero-shot structured-reasoning problem, our method combines multimodal prompting, embedding alignment, and a lightweight pair-refinement strategy, enabling inference over unseen object vocabularies and relation sets. To assess this setting, we formalize an Open-World evaluation protocol that measures performance when no SGG-specific data have been observed either in terms of objects and relations. Experiments on Visual Genome, Open Images V6, and the Panoptic Scene Graph (PSG) dataset demonstrate the capacity of pretrained VLMs to perform relational understanding without task-level training.

Figures

Figures reproduced from arXiv: 2506.08189 by the authors.

Figure 1
Figure 1. Overview of our proposed framework for open-world SGG using VLMs. [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Ablation Study: F1 scores across different (a) α and (b) β values for the Qwen-72B model [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Recall comparison across datasets, models, and methods. [PITH_FULL_IMAGE:figures/full_fig_p013_3.png] view at source ↗
Figures from the paper (7 more)
Figure 5
Figure 5. Figure 5: Example 2 — Initial mapped entities: [ski, light, tree, skier, number, snow, roof]. Irrelevant ob￾jects such as light, number, and roof are removed during filtering. B.5.2. Geometric Pair Refinement Following prior work [6], we adopt the same geometric distance formula…
Figure 6
Figure 6. Figure 6: Example of semantic pair refinement. Given an image [PITH_FULL_IMAGE:figures/full_fig_p014_6.png]
Figure 7
Figure 7. Figure 7: Final scene graph generation setup. Refined object pairs, [PITH_FULL_IMAGE:figures/full_fig_p014_7.png]
Figure 8
Figure 8. Figure 8: Example 1: Qualitative comparison of top object pairs from different methods versus the Ground Truth (GT) pairs for the given [PITH_FULL_IMAGE:figures/full_fig_p015_8.png]
Figure 9
Figure 9. Figure 9: Example 2: Qualitative comparison of top object pairs from different methods versus the Ground Truth (GT) pairs for the given [PITH_FULL_IMAGE:figures/full_fig_p015_9.png]
Figure 10
Figure 10. Figure 10: Example 3: Qualitative comparison of top object pairs from different methods versus the Ground Truth (GT) pairs for the given [PITH_FULL_IMAGE:figures/full_fig_p016_10.png]
Figure 11
Figure 11. Figure 11: The full text of the Semantic Pair Scoring Prompt. [PITH_FULL_IMAGE:figures/full_fig_p017_11.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. GraphVid: Interactive Graph-Controllable Video Generation

    cs.CV 2026-07 conditional novelty 6.0 of 10

    GraphVid controls video generation with user-editable interaction scene graphs, reporting FID/FVD improvements over trajectory- and text-physics baselines using 0.6B trainable parameters.

  2. KG-ViP: Bridging Knowledge Grounding and Visual Perception in Multi-modal LLMs for Visual Question Answering

    cs.CV 2026-01 unverdicted novelty 6.0 of 10

    KG-ViP fuses scene graphs and commonsense graphs via a query-based retrieval-and-fusion pipeline to improve multi-modal LLM performance on visual question answering.

Reference graph

Works this paper leans on

48 extracted references · 31 canonical work pages · cited by 2 Pith papers

  1. [1]

    Qwen-vl: A frontier large vision-language model with versatile abilities.arXiv preprint arXiv:2308.12966, 2023

    Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A frontier large vision-language model with versatile abilities.arXiv preprint arXiv:2308.12966, 2023

    arXiv 2023

  2. [2]

    Gpt4sgg: Synthesizing scene graphs from holistic and region-specific narratives.arXiv preprint arXiv:2312.04314, 2023

    Zuyao Chen, Jinlin Wu, Zhen Lei, Zhaoxiang Zhang, and Changwen Chen. Gpt4sgg: Synthesizing scene graphs from holistic and region-specific narratives.arXiv preprint arXiv:2312.04314, 2023

    arXiv 2023

  3. [3]

    Expanding scene graph boundaries: fully open-vocabulary scene graph generation via visual-concept alignment and retention

    Zuyao Chen, Jinlin Wu, Zhen Lei, Zhaoxiang Zhang, and Chang Wen Chen. Expanding scene graph boundaries: fully open-vocabulary scene graph generation via visual-concept alignment and retention. InEuropean Conference on Com- puter Vision, pages 108–124. Springer, 2024

    work page 2024

  4. [4]

    Reltr: Relation transformer for scene graph generation.IEEE Trans- actions on Pattern Analysis and Machine Intelligence, 45(9): 11169–11183, 2023

    Yuren Cong, Michael Ying Yang, and Bodo Rosenhahn. Reltr: Relation transformer for scene graph generation.IEEE Trans- actions on Pattern Analysis and Machine Intelligence, 45(9): 11169–11183, 2023

    work page 2023

  5. [5]

    Molmo and pixmo: Open weights and open data for state-of-the-art multi- modal models.arXiv preprint arXiv:2409.17146, 2024

    Matt Deitke, Christopher Clark, Sangho Lee, Rohun Tripathi, Yue Yang, Jae Sung Park, Mohammadreza Salehi, Niklas Muennighoff, Kyle Lo, Luca Soldaini, et al. Molmo and pixmo: Open weights and open data for state-of-the-art multi- modal models.arXiv preprint arXiv:2409.17146, 2024

    arXiv 2024

  6. [6]

    Prism-0: A predicate-rich scene graph genera- tion framework for zero-shot open-vocabulary tasks.arXiv preprint arXiv:2504.00844, 2025

    Abdelrahman Elskhawy, Mengze Li, Nassir Navab, and Ben- jamin Busam. Prism-0: A predicate-rich scene graph genera- tion framework for zero-shot open-vocabulary tasks.arXiv preprint arXiv:2504.00844, 2025

    arXiv 2025

  7. [7]

    Simcse: Simple contrastive learning of sentence embeddings.arXiv preprint arXiv:2104.08821, 2021

    Tianyu Gao, Xingcheng Yao, and Danqi Chen. Simcse: Simple contrastive learning of sentence embeddings.arXiv preprint arXiv:2104.08821, 2021

    arXiv 2021

  8. [8]

    Open- vocabulary object detection via vision and language knowl- edge distillation.arXiv preprint arXiv:2104.13921, 2021

    Xiuye Gu, Tsung-Yi Lin, Weicheng Kuo, and Yin Cui. Open- vocabulary object detection via vision and language knowl- edge distillation.arXiv preprint arXiv:2104.13921, 2021

    arXiv 2021

Show all 48 references
  1. [9]

    To- wards open-vocabulary scene graph generation with prompt- 7 based finetuning

    Tao He, Lianli Gao, Jingkuan Song, and Yuan-Fang Li. To- wards open-vocabulary scene graph generation with prompt- 7 based finetuning. InEuropean Conference on Computer Vision, pages 56–73. Springer, 2022

    2022

  2. [10]

    Scene graph reasoning for visual question answering.arXiv preprint arXiv:2007.01072, 2020

    Marcel Hildebrandt, Hang Li, Rajat Koner, V olker Tresp, and Stephan Günnemann. Scene graph reasoning for visual question answering.arXiv preprint arXiv:2007.01072, 2020

    2007 arXiv

  3. [11]

    Gqa: A new dataset for real-world visual reasoning and compositional question answering

    Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. InProceedings of the IEEE/CVF con- ference on computer vision and pattern recognition, pages 6700–6709, 2019

    2019

  4. [12]

    Enhancing scene graph generation with hierarchical relationships and commonsense knowledge

    Bowen Jiang, Zhijun Zhuang, Shreyas S Shivakumar, and Camillo J Taylor. Enhancing scene graph generation with hierarchical relationships and commonsense knowledge. In 2025 IEEE/CVF Winter Conference on Applications of Com- puter Vision (WACV), pages 8883–8894. IEEE, 2025

    2025

  5. [13]

    Image retrieval using scene graphs

    Justin Johnson, Ranjay Krishna, Larry Stark, Jia Li, David A Shamma, Michael Bernstein, and Li Fei-Fei. Image retrieval using scene graphs. InProceedings of the IEEE conference on computer vision and pattern recognition, pages 3668–3678, 2015

    2015

  6. [14]

    Scene graph generation strategy with co-occurrence knowledge and learnable term frequency.arXiv preprint arXiv:2405.12648, 2024

    Hyeongjin Kim, Sangwon Kim, Dasom Ahn, Jong Taek Lee, and Byoung Chul Ko. Scene graph generation strategy with co-occurrence knowledge and learnable term frequency.arXiv preprint arXiv:2405.12648, 2024

    2024 arXiv

  7. [15]

    Llm4sgg: large language models for weakly supervised scene graph generation

    Kibum Kim, Kanghoon Yoon, Jaehyeong Jeon, Yeonjun In, Jinyoung Moon, Donghyun Kim, and Chanyoung Park. Llm4sgg: large language models for weakly supervised scene graph generation. InProceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition, pages 2830...

    2024

  8. [16]

    Visual genome: Connecting language and vision using crowdsourced dense image annotations.International journal of computer vision, 123:32–73, 2017

    Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalan- tidis, Li-Jia Li, David A Shamma, et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations.International journal of computer...

    2017

  9. [17]

    Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper Uijlings, Ivan Krasin, Jordi Pont-Tuset, Shahab Kamali, Stefan Popov, Matteo Malloci, Alexander Kolesnikov, et al. The open im- ages dataset v4: Unified image classification, object detection, and visual relationship detection...

    1956

  10. [18]

    Gonzalez, Hao Zhang, and Ion Stoica

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. InPro- ceedings of the ACM SIGOPS 29th Symposium on Operating System...

    2023

  11. [19]

    Blip: Bootstrapping language-image pre-training for unified vision- language understanding and generation

    Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre-training for unified vision- language understanding and generation. InInternational conference on machine learning, pages 12888–12900. PMLR, 2022

    2022

  12. [20]

    Blip- 2: Bootstrapping language-image pre-training with frozen image encoders and large language models

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip- 2: Bootstrapping language-image pre-training with frozen image encoders and large language models. InInternational conference on machine learning, pages 19730–19742. PMLR, 2023

    2023

  13. [21]

    Sgtr: End-to- end scene graph generation with transformer

    Rongjie Li, Songyang Zhang, and Xuming He. Sgtr: End-to- end scene graph generation with transformer. Inproceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 19486–19496, 2022

    2022

  14. [22]

    From pixels to graphs: Open-vocabulary scene graph generation with vision-language models

    Rongjie Li, Songyang Zhang, Dahua Lin, Kai Chen, and Xuming He. From pixels to graphs: Open-vocabulary scene graph generation with vision-language models. InProceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 28076–28086, 2024

    2024

  15. [23]

    Gps-net: Graph property sensing network for scene graph generation

    Xin Lin, Changxing Ding, Jinquan Zeng, and Dacheng Tao. Gps-net: Graph property sensing network for scene graph generation. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3746–3753, 2020

    2020

  16. [24]

    Llava-next: Improved reason- ing, ocr, and world knowledge, 2024

    Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Improved reason- ing, ocr, and world knowledge, 2024

    2024

  17. [25]

    Visual instruction tuning.Advances in neural information processing systems, 36, 2024

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning.Advances in neural information processing systems, 36, 2024

    2024

  18. [26]

    Grounding dino: Marrying dino with grounded pre-training for open-set object detection

    Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Qing Jiang, Chunyuan Li, Jianwei Yang, Hang Su, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. InEuropean Con- ference on Computer Vision, pages 38–55. Springer, 2024

    2024

  19. [27]

    Relation-aware hierarchical prompt for open-vocabulary scene graph generation

    Tao Liu, Rongjie Li, Chongyu Wang, and Xuming He. Relation-aware hierarchical prompt for open-vocabulary scene graph generation. InProceedings of the AAAI Con- ference on Artificial Intelligence, pages 5576–5584, 2025

    2025

  20. [28]

    Visual relationship detection with language priors

    Cewu Lu, Ranjay Krishna, Michael Bernstein, and Li Fei- Fei. Visual relationship detection with language priors. In Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11–14, 2016, Proceed- ings, Part I 14, pages 852–869. Springer, 2016

    2016

  21. [29]

    hello gpt-4

    OpenAI. “hello gpt-4”, 2024

    2024

  22. [30]

    Learning transferable visual models from natural language supervi- sion

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervi- sion. InInternational conference on machine learning, p...

    2021

  23. [31]

    Sentence-bert: Sentence embeddings using siamese bert-networks.arXiv preprint arXiv:1908.10084, 2019

    Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert-networks.arXiv preprint arXiv:1908.10084, 2019

    1908 arXiv

  24. [32]

    Learning to compose dynamic tree structures for visual contexts

    Kaihua Tang, Hanwang Zhang, Baoyuan Wu, Wenhan Luo, and Wei Liu. Learning to compose dynamic tree structures for visual contexts. InProceedings of the IEEE/CVF con- ference on computer vision and pattern recognition, pages 6619–6628, 2019

    2019

  25. [33]

    Unbiased scene graph generation from bi- ased training

    Kaihua Tang, Yulei Niu, Jianqiang Huang, Jiaxin Shi, and Hanwang Zhang. Unbiased scene graph generation from bi- ased training. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 3716–3725, 2020

    2020

  26. [34]

    Gemini 1.5: Unlocking mul- 8 timodal understanding across millions of tokens of context

    Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, et al. Gemini 1.5: Unlocking mul- 8 timodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05530, 2024

    2024 arXiv

  27. [35]

    Graph-structured representations for visual question answer- ing

    Damien Teney, Lingqiao Liu, and Anton van Den Hengel. Graph-structured representations for visual question answer- ing. InProceedings of the IEEE conference on computer vision and pattern recognition, pages 1–9, 2017

    2017

  28. [36]

    Structured sparse r-cnn for direct scene graph generation

    Yao Teng and Limin Wang. Structured sparse r-cnn for direct scene graph generation. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19437–19446, 2022

    2022

  29. [37]

    Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution.arXiv preprint arXiv:2409.12191, 2024

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution.arXiv preprint arXiv:2409.12191, 2024

    2024 arXiv

  30. [38]

    Scene graph generation by iterative message passing

    Danfei Xu, Yuke Zhu, Christopher B Choy, and Li Fei-Fei. Scene graph generation by iterative message passing. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5410–5419, 2017

    2017

  31. [39]

    Llava-spacesgg: Visual instruct tuning for open-vocabulary scene graph generation with enhanced spatial relations

    Mingjie Xu, Mengyang Wu, Yuzhi Zhao, Jason Chun Lok Li, and Weifeng Ou. Llava-spacesgg: Visual instruct tuning for open-vocabulary scene graph generation with enhanced spatial relations. In2025 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pages 6362–63...

    2025

  32. [40]

    Panoptic scene graph gen- eration

    Jingkang Yang, Yi Zhe Ang, Zujin Guo, Kaiyang Zhou, Wayne Zhang, and Ziwei Liu. Panoptic scene graph gen- eration. InEuropean Conference on Computer Vision, pages 178–196. Springer, 2022

    2022

  33. [41]

    Depth anything v2.Advances in Neural Information Processing Systems, 37: 21875–21911, 2024

    Lihe Yang, Bingyi Kang, Zilong Huang, Zhen Zhao, Xiao- gang Xu, Jiashi Feng, and Hengshuang Zhao. Depth anything v2.Advances in Neural Information Processing Systems, 37: 21875–21911, 2024

    2024

  34. [42]

    Cross-modal rela- tionship inference for grounding referring expressions

    Sibei Yang, Guanbin Li, and Yizhou Yu. Cross-modal rela- tionship inference for grounding referring expressions. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4145–4154, 2019

    2019

  35. [43]

    Visually-prompted language model for fine- grained scene graph generation in an open world

    Qifan Yu, Juncheng Li, Yu Wu, Siliang Tang, Wei Ji, and Yueting Zhuang. Visually-prompted language model for fine- grained scene graph generation in an open world. InProceed- ings of the IEEE/CVF International Conference on Computer Vision, pages 21560–21571, 2023

    2023

  36. [44]

    Open-vocabulary object detection using captions

    Alireza Zareian, Kevin Dela Rosa, Derek Hao Hu, and Shih- Fu Chang. Open-vocabulary object detection using captions. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 14393–14402, 2021

    2021

  37. [45]

    Neural motifs: Scene graph parsing with global context

    Rowan Zellers, Mark Yatskar, Sam Thomson, and Yejin Choi. Neural motifs: Scene graph parsing with global context. arXiv preprint arXiv:1711.06640, 2018

    2018 arXiv

  38. [46]

    Graphical contrastive losses for scene graph parsing

    Ji Zhang, Kevin J Shih, Ahmed Elgammal, Andrew Tao, and Bryan Catanzaro. Graphical contrastive losses for scene graph parsing. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11535– 11543, 2019

    2019

  39. [47]

    Learning to generate language- supervised and open-vocabulary scene graph using pre-trained visual-semantic space

    Yong Zhang, Yingwei Pan, Ting Yao, Rui Huang, Tao Mei, and Chang-Wen Chen. Learning to generate language- supervised and open-vocabulary scene graph using pre-trained visual-semantic space. In2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 291...

    2023

  40. [48]

    There is aXin the image

    Yong Zhang, Yingwei Pan, Ting Yao, Rui Huang, Tao Mei, and Chang-Wen Chen. Learning to generate language- supervised and open-vocabulary scene graph using pre-trained visual-semantic space. InProceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, ...

    2023

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.