Pith. sign in

REVIEW 4 major objections 5 minor 68 references

Hallucinate, Ground, Repeat: A Framework for Generalized Visual Relationship Detection

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that a visual relationship detector can be trained to recognize predicates that never appeared in its training data, using only LLM-hallucinated relationship triplets that are iteratively grounded in image evidence.

desk verdict Useful benchmark and honest writing, but the unseen-generalization claim collapses once you notice the unseen predicates are in the LLM-generated training set. read the letter →

arxiv 2506.05651 v1 pith:7GRQVU4A submitted 2025-06-06 cs.CV

Shanmukha Vellamcheti , Sanjoy Kundu , Sathyanarayanan N. Aakur This is my paper
classification cs.CV
keywords open-worldvisualrelationshipdetectionscenegraphgenerationLLMhallucinationsaspriorsexpectation-maximizationgroundingweaklysupervisedlearningunseenpredicategeneralizationGenomebenchmarkiterativerefinement
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that a visual relationship detector can be trained to recognize both annotated and previously unseen predicates using no human relationship labels at all, by treating a large language model's hallucinated relationship triplets as a structured prior and iteratively grounding them in image evidence. The authors propose EM-Grounding: an LLM (GPT-4o), given only the detected object class names and ordered pairs, proposes up to five plausible predicates per pair, and a transformer-based visual model is trained to predict those triplets from image features; high-confidence predictions on seen predicates are then added back to the training set and the cycle repeats. On a new Visual Genome split with 21 held-out predicates, the method reports mean recall at 50 of 15.9, 13.1, and 11.7 for seen, unseen, and mixed predicate sets, outperforming LLM-only, few-shot, and debiased baselines. The upshot: symbolically rich but visually ungrounded priors can be converted into training signal, offering a path to open-world scene understanding without dense annotation.

What carries the argument

The central object is a semantic relational hypergraph: for each ordered pair of detected objects, the LLM proposes multiple predicates, so each pair carries a set of plausible relations rather than a single edge. This overcomplete symbolic prior, combined with an EM-style refinement loop, carries the argument. The expectation step instantiates the prior with GPT-4o (prompted with object classes, ordered pairs, and a fixed predicate list); the maximization step trains the visual grounding model to maximize log-likelihood of the hallucinated triplets given the image; and the loop closes by adding the model's own high-confidence seen-predicate predictions (above threshold τ = 0.8) to the training set and iterating until convergence (about three iterations). The visual model itself is a decoder-only transformer that builds query embeddings from RoI-pooled visual features and semantic word embeddings of subject and object, and attends over frozen DETR features for global scene context.

What would settle it

Feed the LLM corrupted object labels (e.g., swap 'bottle' with 'bird') while keeping the images fixed, retrain EM-Grounding, and check whether its predicate predictions shift accordingly; if the model's errors track the corrupted hallucinations, then the visual grounding step is not filtering the prior using image evidence. Alternatively, build a test set of images with unusual spatial arrangements where the LLM's most plausible relation for a pair is false, and verify whether the model still predicts the LLM's top-ranked predicate at high confidence.

Watch

Extended reading notes

Core claim

The central discovery is that semantically plausible but visually ungrounded relationship hypotheses—hallucinated by an LLM from object category names alone—can be turned into a training signal for visual relationship detection by an iterative expectation-maximization style loop. In the expectation step, GPT-4o generates a multi-relational hypergraph: for each ordered object pair, up to five ranked predicates with confidence scores, based only on the list of detected classes and the fixed 50-predicate vocabulary. In the maximization step, a decoder-only transformer (adapted from IS-GGT, with DETR image features as global context) is trained to predict those triplets from the image, then its high-confidence within-vocabulary predictions above a threshold are added to the training set, and the model is fine-tuned again. With no ground-truth triplets used at any point, this refinement produces mean recall mR@50 of 15.9 on seen, 13.1 on unseen, and 11.7 on mixed predicates, and on the full Visual Genome test set the model trained on just 475 images and 2.2k hallucinated triplets outperforms fully supervised IMP+ and Neural Motifs. The authors claim that visual grounding of symbolic priors enables generalization to novel relationships.

Load-bearing premise

The load-bearing premise (Sec 4.1) is that relationship triplets hallucinated by the LLM from object class names alone—with no spatial layout, attributes, or image content—are true often enough that training a visual model to predict them improves generalization; if those priors are systematically wrong for a scene, the model learns false relationships, and the refinement loop (Sec 4.3) only adds back high-confidence seen-predicate predictions, amplifying rather than correcting the error.

Editorial extensions

If this is right

  • A relationship classifier can be trained with zero human-annotated predicate labels: the fully grounded EM-Grounding model reaches mR@50 of 15.9 on seen predicates, outperforming the fully supervised GGT baseline's 13.0.
  • The same training loop transfers to held-out predicates: on the unseen-only split, EM-Grounding reaches mR@50 of 13.1, where fully supervised models (GGT, FGPL, HiKER-SGG) collapse to 0.0.
  • On mixed scenes containing both seen and unseen predicates, EM-Grounding reaches mR@50 of 11.7, indicating that refinement supports compositional generalization rather than memorizing the seen predicate set.
  • The framework bridges a large supervision gap on the full Visual Genome test set: despite training on only 475 images and 2.2k triplets versus 57k images and 405k triplets for fully supervised models, the paper reports that it outperforms IMP+ and Neural Motifs and approaches VCTree.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the LLM prior itself carries bias, the grounding loop is more likely to amplify than correct it, since the model maximizes likelihood of the hallucinated triplets and refinement only adds back confident seen-predicate predictions; the paper acknowledges inherited bias but does not measure it.
  • The overcomplete hypergraph may act as a regularizer independently of the LLM's semantic content; a testable extension would replace the LLM with a frequency- or randomness-based prior and compare unseen-set recall to isolate the contribution of semantic plausibility.
  • The same hallucinate-ground-repeat loop could transfer to other structured prediction tasks with scarce annotation, such as video action graphs or affordance prediction, where LLM knowledge of everyday interactions is rich.
  • Feeding spatial layout or attribute information into the LLM prompt, or using a vision-language model to build the prior, could reduce false grounding in unusual scenes; the paper names this direction as future work.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes EM-Grounding, a weakly supervised framework for visual relationship detection that uses GPT-4o to hallucinate candidate (subject, predicate, object) triplets from object category labels alone, then trains an IS-GGT-style visual model on those pseudo-labels (Eq. 2) and iteratively augments the training set with high-confidence seen-predicate predictions (Eq. 3). The authors introduce a new Visual Genome benchmark split into seen, unseen, and mixed predicate subsets and report PredCls/SGCls/SGDet results, with headline PredCls mR@50 values of 15.9 (seen), 13.1 (unseen), and 11.3 (mixed). The central claim is that visual grounding of LLM priors enables generalization to unseen predicates without human relationship labels.

Significance. If the claims were sound, the paper would be a useful contribution: it attacks a real limitation of closed-world SGG, proposes a low-supervision training recipe, releases a structured benchmark with seen/unseen/mixed splits, and compares against several supervision regimes including few-shot and debiased baselines. The iterative hallucinate-and-ground loop is a reasonable design idea, and the ablation of the confidence threshold (Fig. 3b) is a useful practical detail. However, the central generalization claim is currently undercut by a training-set leakage problem: the LLM prompt lists all 50 predicates, including the 21 predicates later classified as unseen, and Eq. 2 trains on the resulting pseudo-labels. Because the 'unseen' evaluation is therefore not blind, the reported unseen-predicate gains do not yet establish generalization to novel relationships.

major comments (4)
  1. [Section 5, Sec 4.1, App. 11.2, Eq. 2] The statement in Section 5 that 'the training set is disjoint from any unseen predicates' is inconsistent with the pipeline described in Sec 4.1 and App 11.2. The GPT-4o prompt in App 11.2 lists all 50 predicates, including the 21 predicates that are later designated unseen, and the hallucinated set T_hat produced from this prompt is used directly as the supervision in Eq. 2. Consequently, the pseudo-label training set contains unseen-predicate triplets, and the unseen-split results in Table 1 measure the model's ability to learn noisy labels for test-vocabulary predicates rather than its ability to visually ground a prior over novel relationships. This is a load-bearing issue for the paper's central claim and must be fixed, ideally by filtering T_hat to the seen predicate set (or by constraining the LLM prompt to seen predicates only) and then re-running all experiments and tables.
  2. [Eq. 2, Eq. 3, Table 1] The training objective in Eq. 2 maximizes the likelihood of positive LLM-generated triplets only; there is no negative or background term, so nothing in the loss directly forces the model to reject plausible-but-absent triplets. The refinement step in Eq. 3 adds only high-confidence seen-predicate predictions and never removes or downweights incorrect hallucinated edges. The empirical evidence that the grounding/refinement loop is what drives the unseen-predicate scores is weak: in Table 1, EM-Grounding (None) reaches 13.1 mR@50 on unseen vs. 12.9 for GPT4o+GGT (which uses the same pseudo-labels without refinement) and 11.4 for the ungrounded GPT-4o baseline, while on mixed mR@50 it is 11.3 vs. 11.4 for GPT4o+GGT. These margins do not support the claim that visual grounding of symbolic priors, rather than the LLM prior itself, produces the unseen-predicate performance.
  3. [App. 11.2, Sec 4.1] The appendix states that 'In all our experiments we only use the predicate with the highest score from these predictions in order to get the triplets.' This contradicts Sec 4.1's description of a multi-relational symbolic prior that returns up to five ranked predicates per object pair and forms a semantic relational hypergraph. If only the highest-scoring predicate per pair is used, then Eq. 2 is trained on single-label triplets, and the claimed hypergraph formulation and the ambiguity-resolution mechanism are not actually exercised. The authors should clarify which protocol was used and, if the hypergraph claim is to be retained, rerun the experiments with the full ranked set of predicates as training targets.
  4. [Sec 5, Table 3, Table 1] The unseen-only evaluation set is very small (167 images, 19 predicates, 361 triplets), and all reported mR@K values are point estimates with no confidence intervals or significance tests. Given that the margins between methods on unseen and mixed mR@50 are around 0.1–0.2 points, the reported ordering of baselines should not be treated as established without variance estimates or repeated-seed results.
minor comments (5)
  1. [Table 1 caption] The caption says EM-Grounding 'consistently outperforms' all weakly- and few-shot baselines, but on mixed mR@50 the Ours (None) row is 11.3 vs. 11.4 for GPT4o+GGT; the word 'consistently' is too strong for this table.
  2. [Table 3] The table uses rows labeled 'Val - Seen', 'Val - Unseen', and 'Val - Mixed', but the text in Section 5 says the evaluation set merges the original VG validation and test splits; the table heading or text should be made consistent.
  3. [Eq. 1] Equation 1 has a typographical oddity ('max θ ; EI,O') and the surrounding sentence 'where {T*, T̂} ∈ T is the space of all candidate triplets' is grammatically unclear; please rewrite this notation.
  4. [Fig. 3] Figure 3 reports ablation results but does not define the axes or include error bars; adding axis labels and indicating the number of runs would help the reader interpret the threshold and iteration curves.
  5. [Section 6, 'Generalized Prediction'] The text claims EM-Grounding 'significantly outperforms all baselines' on the mixed split, but the reported mixed mR@50 is not above GPT4o+GGT; please temper this claim or provide statistical support.

Circularity Check

2 steps flagged · score 6.0 of 10

Unseen predicates are present in the LLM-generated training pseudo-labels, so the 'unseen' evaluation is not blind.

  1. fitted input called prediction [Sec. 4.1 and Sec. 4.2, Eq. (2)]
    "The LLM is prompted with the full list of object categories, all possible ordered object pairs from the image, and a fixed list of 50 predicates (aligning with Visual Genome semantics). ... We supervise the model using only the hallucinated triplets ˆT from the LLM and train it to align symbolic hypotheses with visual content by minimizing: Lalign = − X (s,p,o) ∈ ˆT log Pθ(p | s,o,I)."

    The fixed 50-predicate list in the prompt is the same vocabulary from which the seen/unseen splits are carved, so the 19-21 'held-out' predicates are among the labels the LLM is allowed to propose. The pseudo-label set ˆT is therefore not disjoint from the unseen test vocabulary, and Eq. (2) trains Pθ directly on those unseen-predicate labels. The later 'unseen' evaluation then measures learning of the LLM-generated labels for those predicates, not generalization from a prior over novel predicates; the refinement loop adds only seen high-confidence triplets but never removes the unseen-predicate supervision already present in ˆT.

  2. self definitional [Sec. 5, Experimental Setup]
    "Importantly, the training set is disjoint from any unseen predicates. For evaluation, we merge the original VG validation and test splits to create a combined set of 5,777 images, comprising 40,884 annotated triplets across 50 predicates."

    The sentence defines the training set as the 2,226 human-annotated triplets from 475 images, which indeed contain only the 29 'seen' predicates. But the actual model supervision in Eq. (2) is the LLM hallucinated set ˆT, not the human annotations. Because the LLM prompt explicitly allows all 50 predicates, ˆT includes the predicates that Sec. 5 later labels 'unseen,' so the claim that training is disjoint from unseen predicates is true only for the human labels and false for the pseudo-labels used to train the model. Consequently, the 'unseen' split is not unseen relative to the training signal.

full rationale

The score is 6 because a central 'prediction' — generalization to unseen predicates — reduces by construction to training on LLM pseudo-labels that already contain those unseen predicate labels. The paper's own equations show that Eq. (2) maximizes likelihood over every triplet in ˆT, and Sec. 4.1 shows that ˆT is generated under a prompt restricted to a fixed list of all 50 predicates, which is the same vocabulary later partitioned into seen and unseen. The claimed disjointness of the training set from unseen predicates in Sec. 5 refers only to human-annotated triplets, not to the actual pseudo-label supervision. This is a concrete, quotable reduction rather than a vague concern. The circularity is not driven by self-citation: IS-GGT is used as an architecture, and GPT-4o is an external knowledge source, so those aspects are independently grounded. Even so, the headline unseen-predicate results (e.g., mR@50 = 13.1) are contaminated by the presence of those labels in the training signal, so the paper's central generalization claim is not yet supported by its benchmark.

Assumptions & free parameters 3 free parameters · 5 assumptions · 2 invented entities

The central claim rests on several unvalidated assumptions: that LLM hallucinations over object names are visually relevant, that training on positive pseudo-labels alone teaches a model to reject unsupported relationships, and that the small custom training split is representative. The invented latent variable T* and the semantic relational hypergraph are formal constructs, and the hypergraph is not actually instantiated in the experiments.

free parameters (3)
  • confidence threshold tau = 0.8
    High-confidence seen-predicate triplets are added to the training set only if confidence exceeds tau (Eq. 3); the paper reports that lowering it degrades generalization, so it is tuned empirically.
  • number of refinement iterations = 3
    The refinement loop runs until no new triplets are added, which the paper says converges within 3 iterations; this is a design choice rather than a fitted constant.
  • LLM predicate ranking depth = top-1 (up to 5 requested)
    The method is described as a multi-relational hypergraph with up to five predicates per pair, but Appendix 11.2 says only the highest-scoring predicate is used for triplets, a selection rule that changes the training distribution.
assumptions (5)
  • domain assumption LLM-generated triplets derived only from object category names are a useful prior for visual relationships.
    Sec 4.1 feeds GPT-4o only object labels and ordered pairs, assuming semantic plausibility correlates with visual truth; if false, the model is trained on systematic errors.
  • domain assumption Maximizing likelihood of LLM pseudo-labels learns visual grounding without any negative examples.
    Eq. 2 only maximizes log P_theta for positive pseudo-labeled triplets; there is no loss that penalizes predicting a relation that is visually absent, so grounding is not directly enforced.
  • domain assumption The 475-image and 29-predicate training subset is representative enough to support generalization claims.
    Sec 5 constructs a small training set; no evidence is given that predicate co-occurrence and visual contexts in this subset match the evaluation splits.
  • domain assumption GPT-4o is treated as a reliable external knowledge source; its hallucination distribution is not validated against Visual Genome statistics.
    The paper compares against LLM-only baselines but never measures agreement between LLM proposals and human annotations beyond final task metrics.
  • domain assumption The GGT graph decoder used for edge sampling can be trained from scratch without leaking ground-truth labels.
    Appendix 11.1 says the graph sampling decoder and predicate classifier are trained from scratch using the original pipeline, but the paper does not specify the supervision for the edge decoder; if it uses ground-truth edges, supervision leaks into the weakly supervised setup.
invented entities (2)
  • Latent grounding variable T*
    purpose: Formalizes the subset of LLM-hallucinated triplets that are visually supported and serves as the target of the EM-style objective (Eq. 1).
    T* is never observed or estimated; the E-step is a deterministic LLM output and the M-step trains on all hallucinated triplets, so the latent variable is a formal device without an independent empirical handle.
  • Semantic relational hypergraph
    purpose: Representation of multiple plausible predicates per object pair from the LLM prior.
    The hypergraph is described as a core contribution, but the implementation keeps only the top-1 predicate per pair, so no multi-relational structure is actually trained on.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Hallucinate, Ground, Repeat: A Framework for Generalized Visual Relationship Detection." pith.science (2026). https://pith.science/paper/7GRQVU4A

@misc{pith2026250605651,
  author       = {Pith},
  title        = {Pith review of: Hallucinate, Ground, Repeat: A Framework for Generalized Visual Relationship Detection},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7GRQVU4A}},
  note         = {Machine review of arXiv:2506.05651}
}
read the original abstract

Understanding relationships between objects is central to visual intelligence, with applications in embodied AI, assistive systems, and scene understanding. Yet, most visual relationship detection (VRD) models rely on a fixed predicate set, limiting their generalization to novel interactions. A key challenge is the inability to visually ground semantically plausible, but unannotated, relationships hypothesized from external knowledge. This work introduces an iterative visual grounding framework that leverages large language models (LLMs) as structured relational priors. Inspired by expectation-maximization (EM), our method alternates between generating candidate scene graphs from detected objects using an LLM (expectation) and training a visual model to align these hypotheses with perceptual evidence (maximization). This process bootstraps relational understanding beyond annotated data and enables generalization to unseen predicates. Additionally, we introduce a new benchmark for open-world VRD on Visual Genome with 21 held-out predicates and evaluate under three settings: seen, unseen, and mixed. Our model outperforms LLM-only, few-shot, and debiased baselines, achieving mean recall (mR@50) of 15.9, 13.1, and 11.7 on predicate classification on these three sets. These results highlight the promise of grounded LLM priors for scalable open-world visual understanding.

Figures

Figures reproduced from arXiv: 2506.05651 by the authors.

Figure 1
Figure 1. Overview. (Left) Traditional SGG relies on sparse annotations. (Middle) LLMs can hallucinate relationships, but are ungrounded, multi-relational hypergraphs. (Right) Our method leverages these hallucinations and grounds them through iterative visual alignment. iteratively filters and refines them based on perceptual support. This formulation enables generalized VRD, scaling beyond annotated labels to recognize seen … view at source ↗
Figure 2
Figure 2. Sparse annotations in Visual Genome. Top: Reference images. Bottom: corresponding relational hypergraphs, where solid edges are ground-truth relationships and dashed edges are plausible but unannotated ones. Annotations often ignore valid relations such as “lamp on stand.” commonsense, and world knowledge, e.g., (person, drinking from, bottle) or (bottle, on, table), even if such relationships are unannotated. These… view at source ↗
Figure 3
Figure 3. Ablation studies showing the effect of (a) iterative refinement, (b) threshold (τ ) settings, and (c) annotation sources on mR@50 for seen, unseen, and mixed predicate recognition. 6 Results and Analysis Seen predicate classification [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: Qualitative results on different settings: unseen predicates only (top left), seen only (bottom left), and mixed (top right and bottom right). Correctly predicted groundtruth (GT) edges are in solid black, missed GT edges in solid red, visually correct but unannotated …
Figure 5
Figure 5. Figure 5: Normalized predicate distribution across subsets. 9 Appendix A This supplementary material provides additional details about the dataset and implementation of various baselines. We also provide additional results for scene graph detection, visualizations and analysis o…
Figure 6
Figure 6. Figure 6: Per class recall@20 for (a) seen, (b) unseen, and (c) mixed predicate classification. All the plots also show the histograms of predicate counts in our train set. 18 [PITH_FULL_IMAGE:figures/full_fig_p018_6.png]
Figure 7
Figure 7. Figure 7: Qualitative visualizations on Seen Split showing three examples (rows) comparing image, PredCls, SGCls, and SGDet output graphs. Solid green lines represent accurately predicted groundtruths while solid red lines represent missed predictions. Dashed green lines represe…
Figure 8
Figure 8. Figure 8: Qualitative Visualizations on Unseen Split showing three examples (rows) comparing image, PredCls, SGCls, and SGDet output graphs. Solid green lines represent accurately predicted groundtruths while solid red lines represent missed predictions. Dashed green lines repre…
Figure 9
Figure 9. Figure 9: Qualitative visualizations on Mixed Split showing three examples (rows) comparing image, PredCls, SGCls, and SGDet output graphs. Solid green lines represent accurately predicted groundtruths while solid red lines represent missed predictions. Dashed green lines repres…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

68 extracted references · 51 canonical work pages

  1. [1]

    S. N. Aakur, S. Kundu, and N. Gunti. Knowledge guided learning: Open world egocentric action recognition with zero supervision. Pattern recognition letters, 156:38–45, 2022

    work page 2022

  2. [2]

    Bendale and T

    A. Bendale and T. Boult. Towards open world recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1893–1902, 2015

    work page 1902

  3. [3]

    Brattoli, J

    B. Brattoli, J. Tighe, F. Zhdanov, P. Perona, and K. Chalupka. Rethinking zero-shot video classification: End-to-end training for realistic applications. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4613–4623, 2020

    work page 2020

  4. [4]

    Carion, F

    N. Carion, F. Massa, G. Synnaeve, N. Usunier, A. Kirillov, and S. Zagoruyko. End-to-end object detection with transformers. In European conference on computer vision, pages 213–229. Springer, 2020

    work page 2020

  5. [5]

    Chatterjee, F

    D. Chatterjee, F. Sener, S. Ma, and A. Yao. Opening the vocabulary of egocentric actions. Advances in Neural Information Processing Systems, 36:33174–33187, 2023

    work page 2023

  6. [6]

    T. Chen, W. Yu, R. Chen, and L. Lin. Knowledge-embedded routing network for scene graph generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6163–6171, 2019

    work page 2019

  7. [7]

    Z. Chen, J. Wu, Z. Lei, Z. Zhang, and C. W. Chen. Expanding scene graph boundaries: fully open-vocabulary scene graph generation via visual-concept alignment and retention. In European Conference on Computer Vision, pages 108–124. Springer, 2024

    work page 2024

  8. [8]

    Chiou, H

    M.-J. Chiou, H. Ding, H. Yan, C. Wang, R. Zimmermann, and J. Feng. Recovering the unbiased scene graphs from the biased ones. In Proceedings of the 29th ACM international conference on multimedia, pages 1581–1590, 2021

    work page 2021

Show all 68 references
  1. [9]

    Y . Cong, M. Y . Yang, and B. Rosenhahn. Reltr: Relation transformer for scene graph generation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(9):11169–11183, 2023. 10

    2023

  2. [10]

    Dhamo, A

    H. Dhamo, A. Farshad, I. Laina, N. Navab, G. D. Hager, F. Tombari, and C. Rupprecht. Semantic image manipulation using scene graphs. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5213–5222, 2020

    2020

  3. [11]

    N. Dong, Y . Zhang, M. Ding, and G. H. Lee. Open World DETR: Transformer based Open World Object Detection, Dec. 2022. URLhttp://arxiv.org/abs/2212.02969. arXiv:2212.02969 [cs]

    2022 arXiv

  4. [12]

    Y . Du, F. Wei, Z. Zhang, M. Shi, Y . Gao, and G. Li. Learning to Prompt for Open-V ocabulary Object Detection with Vision-Language Model. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 14064–14073, New Orleans, LA, USA, June

    2022

  5. [13]

    C. L. Folk, R. W. Remington, and J. C. Johnston. Involuntary covert orienting is contingent on attentional control settings. Journal of Experimental Psychology: Human perception and performance, 18(4):1030, 1992

    1992

  6. [14]

    Gu, T.-Y

    X. Gu, T.-Y . Lin, W. Kuo, and Y . Cui. Open-vocabulary Object Detection via Vision and Language Knowledge Distillation, May 2022. URL http://arxiv.org/abs/2104.13921. arXiv:2104.13921 [cs]

    2022 arXiv

  7. [15]

    X. Hu, J. Chen, X. Li, Y . Guo, L. Wen, P. S. Yu, and Z. Guo. Towards understanding factual knowledge of large language models. In The Twelfth International Conference on Learning Representations, 2024

    2024

  8. [16]

    Hurst, A

    A. Hurst, A. Lerer, A. P. Goucher, A. Perelman, A. Ramesh, A. Clark, A. Ostrow, A. Welihinda, A. Hayes, A. Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276, 2024

    2024 arXiv

  9. [17]

    Itti and C

    L. Itti and C. Koch. Computational modelling of visual attention. Nature Reviews Neuroscience, 2(3):194–203, 2001

    2001

  10. [18]

    Johnson, R

    J. Johnson, R. Krishna, M. Stark, L.-J. Li, D. Shamma, M. Bernstein, and L. Fei-Fei. Image retrieval using scene graphs. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3668–3678, 2015

    2015

  11. [19]

    D. P. Kingma. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014

    2014 arXiv

  12. [20]

    Krishna, Y

    R. Krishna, Y . Zhu, O. Groth, J. Johnson, K. Hata, J. Kravitz, S. Chen, Y . Kalantidis, L.-J. Li, D. A. Shamma, et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. International journal of computer vision, 123:32–73, 2017

    2017

  13. [21]

    Kundu and S

    S. Kundu and S. N. Aakur. Is-ggt: Iterative scene graph generation with generative transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 6292–6301, 2023

    2023

  14. [22]

    Kundu, S

    S. Kundu, S. Trehan, and S. N. Aakur. Discovering novel actions from open world egocentric videos with object-grounded visual commonsense reasoning. In European Conference on Computer Vision, pages 39–56. Springer, 2024

    2024

  15. [23]

    Kundu, S

    S. Kundu, S. Vellamchetti, and S. N. Aakur. Probres: Probabilistic jump diffusion for open-world egocentric activity recognition. arXiv preprint arXiv:2504.03948, 2025

    2025

  16. [24]

    J. Li, D. Li, C. Xiong, and S. Hoi. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In International conference on machine learning, pages 12888–12900. PMLR, 2022

    2022

  17. [25]

    L. Li, L. Chen, Y . Huang, Z. Zhang, S. Zhang, and J. Xiao. The devil is in the labels: Noisy label correction for robust scene graph generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18869–18878, 2022

    2022

  18. [26]

    R. Li, S. Zhang, B. Wan, and X. He. Bipartite graph network with adaptive message passing for unbiased scene graph generation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11109–11119, 2021. 11

    2021

  19. [27]

    W. Li, H. Zhang, Q. Bai, G. Zhao, N. Jiang, and X. Yuan. Ppdl: Predicate probability distribution based loss for unbiased scene graph generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19447–19456, 2022

    2022

  20. [28]

    X. Lin, C. Ding, J. Zhang, Y . Zhan, and D. Tao. Ru-net: Regularized unrolling network for scene graph generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19457–19466, 2022

    2022

  21. [29]

    J. Loo, Z. Wu, and D. Hsu. Open scene graphs for open world object-goal navigation. arXiv preprint arXiv:2407.02473, 2024

    2024 arXiv

  22. [30]

    X. Lyu, L. Gao, Y . Guo, Z. Zhao, H. Huang, H. T. Shen, and J. Song. Fine-grained predicates learning for scene graph generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19467–19475, 2022

    2022

  23. [31]

    X. Lyu, L. Gao, P. Zeng, H. T. Shen, and J. Song. Adaptive fine-grained predicates learning for scene graph generation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45 (11):13921–13940, 2023

    2023

  24. [32]

    A. J. Maule and A. C. Edland. The effects of time pressure on human judgement and decision making. In Decision making, pages 203–218. Routledge, 2002

    2002

  25. [33]

    Nguyen, S

    K. Nguyen, S. Tripathi, B. Du, T. Guha, and T. Q. Nguyen. In defense of scene graphs for image captioning. In Proceedings of the IEEE/CVF international conference on computer vision , pages 1407–1416, 2021

    2021

  26. [34]

    S. V . Nuthalapati, R. Chandradevan, E. Giunchiglia, B. Li, M. Kayser, T. Lukasiewicz, and C. Yang. Lightweight visual question answering using scene graphs. In Proceedings of the 30th ACM International Conference on Information & Knowledge Management, pages 3353–3357, 2021

    2021

  27. [35]

    J. Z. Pan, S. Razniewski, J.-C. Kalo, S. Singhania, J. Chen, S. Dietze, H. Jabeen, J. Omeliya- nenko, W. Zhang, M. Lissandrini, et al. Large language models and knowledge graphs: Opportunities and challenges. arXiv preprint arXiv:2308.06374, 2023

    2023 arXiv

  28. [36]

    Petroni, T

    F. Petroni, T. Rocktäschel, P. Lewis, A. Bakhtin, Y . Wu, A. H. Miller, and S. Riedel. Language models as knowledge bases? arXiv preprint arXiv:1909.01066, 2019

    1909 arXiv

  29. [37]

    Radford, J

    A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748–8763. PmLR, 2021

    2021

  30. [38]

    Ravichandran, L

    Z. Ravichandran, L. Peng, N. Hughes, J. D. Griffith, and L. Carlone. Hierarchical representations and explicit memory: Learning effective navigation policies on 3d scene graphs using graph neural networks. In 2022 International Conference on Robotics and Automation (ICRA), pag...

    2022

  31. [39]

    S. Shit, R. Koner, B. Wittmann, J. Paetzold, I. Ezhov, H. Li, J. Pan, S. Sharifzadeh, G. Kaissis, V . Tresp, and B. Menze. Relationformer: A unified framework for image-to-graph generation. In S. Avidan, G. Brostow, M. Cissé, G. M. Farinella, and T. Hassner, editors,Computer V...

    2022

  32. [40]

    L. Shu, H. Xu, and B. Liu. Unseen class discovery in open-world classification. arXiv preprint arXiv:1801.05609, 2018

    2018 arXiv

  33. [41]

    D. J. Simons and C. F. Chabris. Gorillas in our midst: Sustained inattentional blindness for dynamic events. Perception, 28(9):1059–1074, 1999

    1999

  34. [42]

    K. P. Singh, J. Salvador, L. Weihs, and A. Kembhavi. Scene graph contrastive learning for embodied navigation. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 10884–10894, October 2023

    2023

  35. [43]

    Snell, K

    J. Snell, K. Swersky, and R. Zemel. Prototypical networks for few-shot learning. Advances in neural information processing systems, 30, 2017. 12

    2017

  36. [44]

    Suhail, A

    M. Suhail, A. Mittal, B. Siddiquie, C. Broaddus, J. Eledath, G. Medioni, and L. Sigal. Energy- based learning for scene graph generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 13936–13945, June 2021

    2021

  37. [45]

    K. Tang, H. Zhang, B. Wu, W. Luo, and W. Liu. Learning to compose dynamic tree structures for visual contexts. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2019

    2019

  38. [46]

    K. Tang, Y . Niu, J. Huang, J. Shi, and H. Zhang. Unbiased scene graph generation from biased training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020

    2020

  39. [47]

    X. Wang, Q. Yang, Y . Qiu, J. Liang, Q. He, Z. Gu, Y . Xiao, and W. Wang. Knowledgpt: Enhancing large language models with retrieval and storage access on knowledge bases. arXiv preprint arXiv:2308.11761, 2023

    2023 arXiv

  40. [48]

    J. M. Wolfe. Guided search 2.0 a revised model of visual search. Psychonomic Bulletin & Review, 1:202–238, 1994

    1994

  41. [49]

    T. Wu, S. Ge, J. Qin, G. Wu, and L. Wang. Open-vocabulary spatio-temporal action detection. arXiv preprint arXiv:2405.10832, 2024

    2024 arXiv

  42. [50]

    X. Xi, Y . Huang, Z. Zhong, and R. Luo. UMB: Understanding Model Behavior for Open-World Object Detection. 2024

    2024

  43. [51]

    D. Xu, Y . Zhu, C. B. Choy, and L. Fei-Fei. Scene graph generation by iterative message passing. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 5410–5419, 2017

    2017

  44. [52]

    H. Xu, G. Ghosh, P.-Y . Huang, D. Okhonko, A. Aghajanyan, F. Metze, L. Zettlemoyer, and C. Feichtenhofer. Videoclip: Contrastive pre-training for zero-shot video-text understanding. arXiv preprint arXiv:2109.14084, 2021

    2021 arXiv

  45. [53]

    S. Yan, C. Shen, Z. Jin, J. Huang, R. Jiang, Y . Chen, and X.-S. Hua. Pcpl: Predicate-correlation perception learning for unbiased scene graph generation. In Proceedings of the 28th ACM international conference on multimedia, pages 265–273, 2020

    2020

  46. [54]

    J. Yang, J. Lu, S. Lee, D. Batra, and D. Parikh. Graph r-cnn for scene graph generation. In Proceedings of the European conference on computer vision (ECCV), pages 670–685, 2018

    2018

  47. [55]

    J. Yu, Y . Chai, Y . Wang, Y . Hu, and Q. Wu. Cogtree: Cognition tree loss for unbiased scene graph generation. arXiv preprint arXiv:2009.07526, 2020

    2009 arXiv

  48. [56]

    Q. Yu, J. Li, Y . Wu, S. Tang, W. Ji, and Y . Zhuang. Visually-prompted language model for fine- grained scene graph generation in an open world. InProceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 21560–21571, October 2023

    2023

  49. [57]

    Zareian, S

    A. Zareian, S. Karaman, and S.-F. Chang. Bridging knowledge graphs to generate scene graphs. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XXIII 16, pages 606–623. Springer, 2020

    2020

  50. [58]

    Zellers, M

    R. Zellers, M. Yatskar, S. Thomson, and Y . Choi. Neural motifs: Scene graph parsing with global context. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5831–5840, 2018

    2018

  51. [59]

    Zhang, Y

    A. Zhang, Y . Yao, Q. Chen, W. Ji, Z. Liu, M. Sun, and T.-S. Chua. Fine-grained scene graph generation with data transfer. In European conference on computer vision , pages 409–424. Springer, 2022

    2022

  52. [60]

    Zhang, S

    C. Zhang, S. Stepputtis, J. Campbell, K. Sycara, and Y . Xie. Hiker-sgg: Hierarchical knowledge enhanced robust scene graph generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 28233–28243, 2024. 13

    2024

  53. [61]

    Zhang, J

    Q. Zhang, J. Dong, H. Chen, D. Zha, Z. Yu, and X. Huang. Knowgpt: Knowledge graph based prompting for large language models. Advances in Neural Information Processing Systems, 37: 6052–6080, 2024

    2024

  54. [62]

    Zhao and H

    S. Zhao and H. Xu. Less is more: Toward zero-shot local scene graph generation via foundation models. arXiv preprint arXiv:2310.01356, 2023

    2023 arXiv

  55. [63]

    Zhong, J

    Y . Zhong, J. Shi, J. Yang, C. Xu, and Y . Li. Learning to generate scene graph from natural language supervision. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 1823–1834, October 2021

    2021

  56. [64]

    unknowns

    Z. Zhou, Z. Zhu, H. Caesar, and M. Shi. Openpsg: Open-set panoptic scene graph generation via large multimodal models. In European Conference on Computer Vision, pages 199–215. Springer, 2024. 14 Split #Images #Predicates #Triplets Train 475 29 2,226Val - Seen 4,461 29 28,322V...

    2024

  57. [66]

    (’human’,’tree’); ’under’,0.9; ’near’,0.9; ’in front of’,0.8; ’behind ’,0.8; ’looking at’,0.6

  58. [67]

    (’fruit’,’tree’); ’growing on’,0.9; ’hanging from’,0.9; ’attached to’,0.9; ’under’,0.8; ’near’,0.8

  59. [68]

    For fair comparison we use the GGT graph decoder to obtain the edges and then filtering the GPT4o triplets to include only these edges for computing metrics

    (’tree’,’fruit’); ’over’,0.9; ’near’,0.9; ’attached to’,0.9; ’behind’,0.6; ’across’,0.5 In all our experiments we only use the predicate with the highest score from these predictions in order to get the triplets. For fair comparison we use the GGT graph decoder to obtain the e...

  60. [2022]

    ISBN 978-1-6654-6946-3

    IEEE. ISBN 978-1-6654-6946-3. doi: 10.1109/CVPR52688.2022.01369. URL https: //ieeexplore.ieee.org/document/9878606/

    2022

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.