Pith. sign in

REVIEW 5 major objections 5 minor 76 references

Relation-aware Hierarchical Prompt for Open-vocabulary Scene Graph Generation

T0 review · 5 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims that replacing fixed predicate prompts with hierarchical entity-aware and region-aware text prompts, filtered by a vision-language model, consistently improves open-vocabulary scene graph generation on Visual Genome and…

desk verdict RAHP is a plausible and well-controlled empirical extension of hierarchical prompting for open-vocabulary scene graph generation, but the paper's own ablation shows the region-aware prompts only work when paired with dynamic selection, and the claimed grounding mechanism is not directly validated. read the letter →

arxiv 2412.19021 v2 pith:XDEWEZXS submitted 2024-12-26 cs.CV cs.AI

Tao Liu , Rongjie Li , Chongyu Wang , Xuming He This is my paper · ORCID
classification cs.CVcs.AI
keywords open-vocabularyscenegraphgenerationvisualrelationshipdetectionvision-languagemodelshierarchicaltextpromptsentityclusteringregion-awarepromptminingdynamicselectionzero-shotrelationclassification
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that open-vocabulary scene graph generation is held back by fixed, single-form text prompts for predicates, and that richer text representations can close the gap. To test this, the authors build RAHP, a framework that generates two levels of prompts: entity-aware prompts that combine predicates with clustered "super entities," and region-aware prompts that use a large language model to describe the part-level visual interactions between subject and object. A vision-language model then filters the region prompts against the image content before scores from both levels are aggregated. The paper reports that this design consistently beats prior open-vocabulary systems on Visual Genome and Open Images v6, especially on novel predicates. If correct, the contribution is a lightweight text-side upgrade: any existing scene-graph predictor can gain open-vocabulary ability by swapping in richer, image-filtered prompts.

What carries the argument

The carrying object is the hierarchical prompt stack plus a top-k filter. Entity clustering (WordNet grouping, VLM-embedding K-means, LLM naming) compresses the cubic triplet space into super-entity pairs so entity-aware prompts are feasible. A large language model then mines region-aware descriptions that name the part-level contact between subject and object. A vision-language model scores these region prompts against the union bounding-box crop, keeps the top k, and the final predicate logit is a weighted mixture of the entity-aware and region-aware similarities. The hyperparameters k and α control how many region prompts survive and how much they weigh.

What would settle it

If the top-k region prompts selected by vision-language matching are no more predictive of the true predicate than k prompts drawn at random from the same pool, then the dynamic selection mechanism is not contributing; that comparison is directly computable on Visual Genome using the paper's own pipeline.

Watch

Extended reading notes

Core claim

RAHP's central claim is that the quality of open-vocabulary relation prediction is determined less by the visual encoder than by the text representation it must match. The framework keeps the vision-language model's text encoder frozen and instead expands the text side: predicates are combined with entity clusters to form entity-aware prompts, and an LLM decomposes subject and object into parts to generate region-aware descriptions such as "human buttocks are making contact with the seat of the furniture." During inference, the union region of the subject-object pair is used to rank the region prompts, keep the top k, and weight them into the final predicate score. On Visual Genome and Open Images v6, the paper reports consistent state-of-the-art results, with the largest gains on novel (unseen) predicates.

Load-bearing premise

The whole mechanism presumes that region-aware text descriptions written by an LLM without seeing an image land close, in the vision-language model's embedding space, to the visual features of the actual image region, so that top-k selection keeps helpful prompts rather than noise.

Editorial extensions

If this is right

  • Any one-stage or two-stage scene graph model can adopt RAHP by replacing its fixed predicate classifier, as demonstrated with SGTR+, PE-NET, OvSGTR, and VS3.
  • Novel (unseen) predicate recall improves substantially; for example, PE-NET+RAHP raises novel mR@100 from 19.32 to 23.73 on Visual Genome PredCLS over a fixed-prompt baseline.
  • Filtering region prompts with the vision-language model is necessary: using all region prompts hurts novel predicate performance, while keeping only the top-1 hurts diversity.
  • The method transfers to fully open-vocabulary detection+relation settings, improving novel relation recall on OVD+R-SGG.
  • Entity clustering to 30 super entities on Visual Genome recovers nearly all the benefit of using all 1,125,000 full triplets.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the text side is frozen and only prompts change, the framework could port to other open-vocabulary tasks such as object detection, segmentation, or human-object interaction, where part-level region descriptions can be mined without visual grounding.
  • The paper's own ablation hints that the region descriptions are the main source of novel-relation gains; a direct test would be to see whether paraphrased or ground-truth-grounded descriptions change performance, which would isolate whether the LLM's wording or its part decomposition carries the signal.
  • The weakest link is likely the diversity of LLM-generated descriptions; if one could measure visual grounding of each region prompt or add image-conditioned prompt generation, the top-k selection might become even more effective, and the method might extend beyond the datasets tested.
  • Entity clustering granularity is a latent trade-off: too few super entities wash out object identity, too many reintroduce the triplet-space explosion; this suggests an adaptive clustering scheme as a natural next step.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper proposes RAHP, a prompt-construction framework for CLIP-based open-vocabulary scene graph generation (OV-SGG). It clusters entity categories into super entities, generates entity-aware prompts by combining super entities with predicates, and uses an LLM to produce region-aware part-level descriptions for each super-entity triplet. A VLM-guided dynamic selection step chooses the top-k region prompts per union-box visual feature, and predicate scores are obtained by a weighted average of entity-aware and region-aware similarities. Experiments on Visual Genome and Open Images v6 under OVR-SGG and OVD+R-SGG settings report consistent gains over fixed-prompt counterparts, with ablations decomposing the contributions of the three components. Code is released.

Significance. The paper addresses a real bottleneck in OV-SGG: fixed text prompts limit CLIP-based relation classification. The entity-clustering trick is a practical way to control the combinatorial explosion of triplets, and the idea of generating region-level descriptions with an LLM is timely. The controlled within-backbone comparisons (PE-NET+RAHP vs. PE-NET+p, SGTR†+RAHP vs. SGTR†+p) show credible gains on novel predicates, the ablation isolates components, and the held-out novel predicate evaluation is external to the prompt-generation process. If the mechanism is validated, RAHP would be a useful and transferable recipe for other VLM-based open-vocabulary tasks. However, the empirical support for "consistently state-of-the-art" is weakened by single-run results, partly confounded cross-backbone comparisons, and the absence of direct evidence that the region prompts selected by Eq. (3) are actually grounded in the visual interaction.

major comments (5)
  1. [§5.2, Tables 1 and 2] The abstract and Sec. 6 state that RAHP "consistently achieves state-of-the-art performance", but the cross-method comparisons in Tables 1 and 2 are not controlled. The headline improvement over PGSG (e.g., 21.56 points in total R@100 on OIV6) compares models with different backbones and detectors (ViT DETR* vs. R-101 DETR, R-101 Faster R-CNN) and different predicate splits. The controlled gains (e.g., PE-NET+RAHP vs. PE-NET+p) are credible, but the manuscript should either soften the SOTA claim or present like-for-like comparisons with published methods under identical backbones and splits.
  2. [§4.4, Eq. (3) and Table 4] The central novelty claim is that region-aware prompts capture fine-grained visual interactions, but no experiment verifies that the prompts selected by Eq. (3) are visually grounded. Table 4 row 3 shows that adding region-aware prompts without dynamic selection drops novel R@50 from 6.15 to 2.38, so the entire benefit of the region-aware component is attributed to selection. Selection is based on CLIP similarity with the union box, which may be driven by generic part/object recognition rather than predicate-specific interaction. The paper should provide direct evidence—e.g., analysis of selected prompts per predicate, comparison with random or static prompt subsets, or a region-grounding metric—before attributing the gain to "fine-grained visual interactions".
  3. [Tables 1–3] All reported numbers are single-run point estimates with no error bars or significance tests. Some controlled margins are small—e.g., SGTR†+RAHP vs. SGTR†+p on VG: total R@50 39.92 vs. 39.48 and base R@50 41.29 vs. 40.78—so the "consistently" claim requires multiple seeds (at least 3) and reported mean ± std, especially for novel-predicate metrics.
  4. [§4.4, Eqs. (3)–(5)] The definition of the selection score is ambiguous. T^r_j is defined in Sec. 4.2 as an element of R^{C_p × N^r_j × d}, but Eq. (3) computes φ(U, T^r_j) with φ defined in Eq. (1) for 2D inputs, and writes the output as S^se_j ∈ R^{N × N^r_j}. It is unclear whether selection is performed separately per predicate or shared across all C_p predicates, and how Eq. (5) maps the selected 3D tensor to the 2D score S^r_j. Please clarify the tensor shapes and indexing; this is essential for reproducibility of the dynamic-selection mechanism.
  5. [§5.3, Table 3] In the fully open-vocabulary OVD+R-SGG setting, OvSGTR+RAHP improves novel predicate R@100 from 11.19 to 16.46 but decreases novel object R@50/R@100 from 14.37/17.44 to 12.45/15.38, and total R@100 increases only from 16.36 to 16.52. The paper mentions this trade-off but does not analyze it. Since the framework is proposed for open-vocabulary scene graph generation, the reported degradation on novel objects should be investigated or clearly qualified before claiming "state-of-the-art generalization" in this setting.
minor comments (5)
  1. [References] The reference list contains entries unrelated to the paper (Alpher et al. 2002–2005; Anonymous 2024a,b) that appear to be leftover template text; they should be removed.
  2. [Appendix E.7] The sentence "The total cost of generating region-aware prompts ... is [insert cost here]" is an unfinished placeholder and must be completed or deleted.
  3. [§5.1] The implementation details refer to "PR-NET" but the rest of the paper and Table 4 use "PE-NET"; please correct the typo.
  4. [Table 4 and Table 7] There are typos in the table headers: "Realtion" in Table 4 and "CPT-4o-mini" in Table 7 should be "Relation" and "GPT-4o-mini".
  5. [Appendix C.1] The statistics "20.32 unique objects and 7.58 unique relations" are unclear; please specify whether these are per triplet, per predicate, or across the whole prompt set.

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity: held-out novel predicate evaluation and frozen CLIP/LLM prompts keep the empirical claims independent; the only self-citations are backbone/loss reuse and are not load-bearing.

full rationale

The paper is an empirical alignment system rather than a derivation-from-principles, so the default circularity checks mostly do not apply. The central claims are evaluated on held-out novel predicate splits (VG and OIV6) that are external to the prompt-generation procedure: the LLM generates region descriptions from predicate and super-entity names without access to test images, and the CLIP text encoder is frozen. Novel predicates are not fitted on training data, and test recall is measured against externally defined categories. Equations (3)-(7) combine a CLIP-based prompt-selection step with cosine-similarity predicate scoring; although the same frozen CLIP backbone is reused for both selection and scoring, the inputs are different (union-region features U versus relation features R, and different text embedding sets), so no output is equal to an input by construction. The self-citations, notably SGTR+ as a backbone and the multi-task loss design, are standard engineering reuse rather than load-bearing premises, and no uniqueness theorem is imported from the authors' prior work. The ablation in Table 4 and the stated limitations (limited diversity of LLM-generated region descriptions, entity-clustering granularity) are honest empirical caveats rather than circular reasoning. The reference list contains placeholder 'Frobnication' entries and Appendix E.7 contains an '[insert cost here]' placeholder; these are completeness defects that do not create a circular reduction. Overall, the empirical pipeline is self-contained against external benchmarks, and the minor self-citation is not load-bearing, consistent with a low circularity score.

Assumptions & free parameters 4 free parameters · 4 assumptions · 1 invented entities

The central claim depends on CLIP's pretrained alignment, on LLM text quality, on the chosen super-entity taxonomy, and on validation-tuned hyperparameters. No new physical entities are introduced; however, the super-entity taxonomy is an invented intermediate representation whose quality is not independently validated.

free parameters (4)
  • Number of super entities M = 30 for VG, 53 for OIV6
    Chosen via validation sweeps (Appendix E.6); controls granularity of triplet-aware prompts.
  • Dynamic selection top-k = 3
    Validation sweep in Appendix E.4; k=3 balances noise and diversity.
  • Aggregation weight alpha = 0.25
    Validation sweep in Appendix E.3; controls entity versus region score balance.
  • Loss weights lambda1, lambda2, lambda3 = 2, 1, 20
    lambda3=20 chosen from Appendix E.5; lambda1 and lambda2 are fixed.
assumptions (4)
  • domain assumption CLIP frozen encoders provide a reliable shared space for image-text matching.
    All scores are computed in CLIP space (Eqs. 1-5), and the distillation loss (Eq. 9) assumes relation features can be aligned to CLIP visual features.
  • ad hoc to paper LLM-generated region descriptions are semantically grounded without being generated from images.
    Region-aware mining (Sec. 4.2, App. C) relies on GPT-3.5 part decomposition; Sec. 6 admits diversity limits, and no visual grounding check is provided.
  • domain assumption Entity clustering preserves the information needed for relation prediction.
    WordNet/K-means/LLM clustering (App. B) collapses 150/602 entities to 30/53 super-entities; the paper's own limitation (1) notes fine distinctions can be lost.
  • domain assumption The base/novel splits and Recall@K metrics faithfully measure open-vocabulary generalization.
    The evaluation follows the splits of Epic, OvSGTR, and PGSG; if these splits or metrics are not representative, the SOTA conclusion would be weakened.
invented entities (1)
  • Super-entity categories (e.g., 'limb and appendage', 'seating furniture')
    purpose: Collapse the entity space so triplet-aware prompts become computationally tractable.
    The super-entity labels are produced by clustering and LLM naming (App. B); they are a paper-specific taxonomy with no external falsifiable handle, and their granularity is tuned on validation.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Relation-aware Hierarchical Prompt for Open-vocabulary Scene Graph Generation." pith.science (2026). https://pith.science/paper/XDEWEZXS

@misc{pith2026241219021,
  author       = {Pith},
  title        = {Pith review of: Relation-aware Hierarchical Prompt for Open-vocabulary Scene Graph Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XDEWEZXS}},
  note         = {Machine review of arXiv:2412.19021}
}
read the original abstract

Open-vocabulary Scene Graph Generation (OV-SGG) overcomes the limitations of the closed-set assumption by aligning visual relationship representations with open-vocabulary textual representations. This enables the identification of novel visual relationships, making it applicable to real-world scenarios with diverse relationships. However, existing OV-SGG methods are constrained by fixed text representations, limiting diversity and accuracy in image-text alignment. To address these challenges, we propose the Relation-Aware Hierarchical Prompting (RAHP) framework, which enhances text representation by integrating subject-object and region-specific relation information. Our approach utilizes entity clustering to address the complexity of relation triplet categories, enabling the effective integration of subject-object information. Additionally, we utilize a large language model (LLM) to generate detailed region-aware prompts, capturing fine-grained visual interactions and improving alignment between visual and textual modalities. RAHP also introduces a dynamic selection mechanism within Vision-Language Models (VLMs), which adaptively selects relevant text prompts based on the visual content, reducing noise from irrelevant prompts. Extensive experiments on the Visual Genome and Open Images v6 datasets demonstrate that our framework consistently achieves state-of-the-art performance, demonstrating its effectiveness in addressing the challenges of open-vocabulary scene graph generation. The code is available at: https://github.com/Leon022/RAHP

Figures

Figures reproduced from arXiv: 2412.19021 by the authors.

Figure 1
Figure 1. An illustration of RAHP for OV-SGG. RAHP gen [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. An overview of hierarchical prompt generation: (a) Predicates combine with super entities to create entity-aware [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. An overview of RAHP. (a) Visual Relation Extraction Module: The process begins with extracting relation proposals [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (4 more)
Figure 5
Figure 5. Figure 5: Impact of Hyper-Parameter k on RAHP Perfor￾mance in the VG validation set. weight of the region-aware prompt in predicate score aggre￾gation. The SGG model used in this experiment is PE-NET. As shown in [PITH_FULL_IMAGE:figures/full_fig_p013_5.png]
Figure 4
Figure 4. Figure 4: Impact of Hyper-Parameter α on RAHP Perfor￾mance in the VG validation set. SGTR† . The results indicate that substituting the fixed classi￾fier with RAHP’s open-vocabulary classifier enhances both re￾call and mean recall in PE-NET and SGTR† . For VS3 , the per￾formance…
Figure 6
Figure 6. Figure 6: Impact of Distillation Loss Weight λ3 on RAHP Performance in the VG validation set. novel relationships, there is a threshold beyond which further increases in λ3 may begin to degrade the model’s overall effectiveness by disproportionately affecting base predicate perf…
Figure 8
Figure 8. Figure 8: Qualitative Results of OvSGTR and RAHP on the SGDet task and VG dataset. [PITH_FULL_IMAGE:figures/full_fig_p016_8.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

76 extracted references · 50 canonical work pages

  1. [1]

    , " * write output.state after.block = add.period write newline

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION init.state.consts #0 'before.a...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Alpher, F. 2002. Frobnication. IEEE TPAMI, 12(1): 234--778

    work page 2002

  4. [4]

    Alpher, F.; and Fotheringham-Smythe, F. 2003. Frobnication revisited. Journal of Foo, 13(1): 234--778

    work page 2003

  5. [5]

    Alpher, F.; Fotheringham-Smythe, F.; and Gamow, F. 2004. Can a machine frobnicate? Journal of Foo, 14(1): 234--778

    work page 2004

  6. [6]

    Alpher, F.; and Gamow, F. 2005. Can a computer frobnicate? In CVPR, 234--778

    work page 2005

  7. [7]

    Anonymous. 2024 a . The frobnicatable foo filter. ECCV submission ID 00324, supplied as supplemental material 00324.pdf

    work page 2024

  8. [8]

    Anonymous. 2024 b . Frobnication tutorial. Supplied as supplemental material tr.pdf

    work page 2024

Show all 76 references
  1. [9]

    M.; and Hassner, T., eds

    Avidan, S.; Brostow, G.; Cissé, M.; Farinella, G. M.; and Hassner, T., eds. 2022. Computer Vision -- ECCV 2022. Springer

    2022

  2. [10]

    D.; Dhariwal, P.; Neelakantan, A.; Shyam, P.; Sastry, G.; Askell, A.; et al

    Brown, T.; Mann, B.; Ryder, N.; Subbiah, M.; Kaplan, J. D.; Dhariwal, P.; Neelakantan, A.; Shyam, P.; Sastry, G.; Askell, A.; et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33: 1877--1901

    2020

  3. [11]

    Bucher, M.; Vu, T.-H.; Cord, M.; and P \'e rez, P. 2019. Zero-shot semantic segmentation. Advances in Neural Information Processing Systems, 32

    2019

  4. [12]

    Carion, N.; Massa, F.; Synnaeve, G.; Usunier, N.; Kirillov, A.; and Zagoruyko, S. 2020. End-to-end object detection with transformers. In European conference on computer vision, 213--229. Springer

    2020

  5. [13]

    Chen, S.; Jin, Q.; Wang, P.; and Wu, Q. 2020. Say as you wish: Fine-grained control of image caption generation with abstract scene graphs. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 9962--9971

    2020

  6. [14]

    Chen, Z.; Wu, J.; Lei, Z.; Zhang, Z.; and Chen, C. 2023. Expanding Scene Graph Boundaries: Fully Open-vocabulary Scene Graph Generation via Visual-Concept Alignment and Retention. arXiv preprint arXiv:2311.10988

    2023 arXiv

  7. [15]

    Gao, K.; Chen, L.; Zhang, H.; Xiao, J.; and Sun, Q. 2023. Compositional prompt tuning with motion cues for open-vocabulary video relation detection. arXiv preprint arXiv:2302.00268

    2023 arXiv

  8. [16]

    Ge, Y.; Ren, J.; Gallagher, A.; Wang, Y.; Yang, M.-H.; Adam, H.; Itti, L.; Lakshminarayanan, B.; and Zhao, J. 2023. Improving zero-shot generalization and robustness of multi-modal models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 11...

    2023

  9. [17]

    Ghiasi, G.; Gu, X.; Cui, Y.; and Lin, T.-Y. 2022. Scaling open-vocabulary image segmentation with image-level labels. In European Conference on Computer Vision, 540--557. Springer

    2022

  10. [18]

    Gothoskar, N.; Cusumano-Towner, M.; Zinberg, B.; Ghavamizadeh, M.; Pollok, F.; Garrett, A.; Tenenbaum, J.; Gutfreund, D.; and Mansinghka, V. 2021. 3DP3: 3D scene perception via probabilistic programming. Advances in Neural Information Processing Systems, 34: 9600--9612

    2021

  11. [19]

    Gu, J.; Joty, S.; Cai, J.; Zhao, H.; Yang, X.; and Wang, G. 2019 a . Unpaired image captioning via scene graph alignments. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 10323--10332

    2019

  12. [20]

    Gu, J.; Zhao, H.; Lin, Z.; Li, S.; Cai, J.; and Ling, M. 2019 b . Scene graph generation with external knowledge and image reconstruction. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 1969--1978

    2019

  13. [21]

    Gu, X.; Lin, T.-Y.; Kuo, W.; and Cui, Y. 2021. Open-vocabulary object detection via vision and language knowledge distillation. arXiv preprint arXiv:2104.13921

    2021 arXiv

  14. [22]

    He, K.; Zhang, X.; Ren, S.; and Sun, J. 2016. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, 770--778

    2016

  15. [23]

    He, T.; Gao, L.; Song, J.; and Li, Y.-F. 2022. Towards open-vocabulary scene graph generation with prompt-based finetuning. In European Conference on Computer Vision, 56--73. Springer

    2022

  16. [24]

    Jia, C.; Yang, Y.; Xia, Y.; Chen, Y.-T.; Parekh, Z.; Pham, H.; Le, Q.; Sung, Y.-H.; Li, Z.; and Duerig, T. 2021. Scaling up visual and vision-language representation learning with noisy text supervision. In International conference on machine learning, 4904--4916. PMLR

    2021

  17. [25]

    Johnson, J.; Krishna, R.; Stark, M.; Li, L.-J.; Shamma, D.; Bernstein, M.; and Fei-Fei, L. 2015. Image retrieval using scene graphs. In Proceedings of the IEEE conference on computer vision and pattern recognition, 3668--3678

    2015

  18. [26]

    Kamath, A.; Singh, M.; LeCun, Y.; Synnaeve, G.; Misra, I.; and Carion, N. 2021. Mdetr-modulated detection for end-to-end multi-modal understanding. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 1780--1790

    2021

  19. [27]

    S.; Reid, M.; Matsuo, Y.; and Iwasawa, Y

    Kojima, T.; Gu, S. S.; Reid, M.; Matsuo, Y.; and Iwasawa, Y. 2022. Large language models are zero-shot reasoners. Advances in neural information processing systems, 35: 22199--22213

    2022

  20. [28]

    A.; et al

    Krishna, R.; Zhu, Y.; Groth, O.; Johnson, J.; Hata, K.; Kravitz, J.; Chen, S.; Kalantidis, Y.; Li, L.-J.; Shamma, D. A.; et al. 2017. Visual genome: Connecting language and vision using crowdsourced dense image annotations. International journal of computer vision, 123: 32--73

    2017

  21. [29]

    Kuznetsova, A.; Rom, H.; Alldrin, N.; Uijlings, J.; Krasin, I.; Pont-Tuset, J.; Kamali, S.; Popov, S.; Malloci, M.; Kolesnikov, A.; et al. 2020. The open images dataset v4: Unified image classification, object detection, and visual relationship detection at scale. Internationa...

    2020

  22. [30]

    Lee, S.; Kim, J.-W.; Oh, Y.; and Jeon, J. H. 2019. Visual question answering over scene graph. In 2019 First International Conference on Graph Computing (GC), 45--50. IEEE

    2019

  23. [31]

    Li, J.; Li, D.; Savarese, S.; and Hoi, S. 2023. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. arXiv preprint arXiv:2301.12597

    2023 arXiv

  24. [32]

    Li, J.; Wang, Y.; Guo, X.; Yang, R.; and Li, W. 2024 a . Leveraging Predicate and Triplet Learning for Scene Graph Generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 28369--28379

    2024

  25. [33]

    Li, L.; Xiao, J.; Chen, G.; Shao, J.; Zhuang, Y.; and Chen, L. 2024 b . Zero-shot visual relation detection via composite visual cues from large language models. Advances in Neural Information Processing Systems, 36

    2024

  26. [34]

    Li, R.; Zhang, S.; and He, X. 2024. SGTR+: End-to-end Scene Graph Generation with Transformer. arXiv:2401.12835

    2024 arXiv

  27. [35]

    Li, R.; Zhang, S.; Lin, D.; Chen, K.; and He, X. 2024 c . From Pixels to Graphs: Open-Vocabulary Scene Graph Generation with Vision-Language Models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 28076--28086

    2024

  28. [36]

    Li, X.; Chen, L.; Ma, W.; Yang, Y.; and Xiao, J. 2022. Integrating object-aware and interaction-aware knowledge for weakly supervised scene graph generation. In Proceedings of the 30th ACM International Conference on Multimedia, 4204--4213

    2022

  29. [37]

    Li, Y.; Pan, Y.; Chen, J.; Yao, T.; and Mei, T. 2021. X-modaler: A versatile and high-performance codebase for cross-modal analytics. In Proceedings of the 29th ACM International Conference on Multimedia, 3799--3802

    2021

  30. [38]

    Liang, F.; Wu, B.; Dai, X.; Li, K.; Zhao, Y.; Zhang, H.; Zhang, P.; Vajda, P.; and Marculescu, D. 2023. Open-vocabulary semantic segmentation with mask-adapted clip. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 7061--7070

    2023

  31. [39]

    Liao, Y.; Zhang, A.; Lu, M.; Wang, Y.; Li, X.; and Liu, S. 2022. Gen-vlkt: Simplify association and enhance interaction understanding for hoi detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 20123--20132

    2022

  32. [40]

    Lin, T.-Y.; Goyal, P.; Girshick, R.; He, K.; and Doll \'a r, P. 2017. Focal loss for dense object detection. In Proceedings of the IEEE international conference on computer vision, 2980--2988

    2017

  33. [41]

    L.; Ricci, E.; Csurka, G.; and Volpi, R

    Liu, M.; Hayes, T. L.; Ricci, E.; Csurka, G.; and Volpi, R. 2024. SHiNe: Semantic Hierarchy Nexus for Open-vocabulary Object Detection. arXiv preprint arXiv:2405.10053

    2024 arXiv

  34. [42]

    Liu, Z.; Lin, Y.; Cao, Y.; Hu, H.; Wei, Y.; Zhang, Z.; Lin, S.; and Guo, B. 2021. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference on computer vision, 10012--10022

    2021

  35. [43]

    Ma, Z.; Luo, G.; Gao, J.; Li, L.; Chen, Y.; Wang, S.; Zhang, C.; and Hu, W. 2022. Open-vocabulary one-stage detection with hierarchical visual-language knowledge distillation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 14074--14083

    2022

  36. [44]

    Menon, S.; and Vondrick, C. 2022. Visual classification via description from large language models. arXiv preprint arXiv:2210.07183

    2022 arXiv

  37. [45]

    Miller, G. A. 1995. WordNet: a lexical database for English. Communications of the ACM, 38(11): 39--41

    1995

  38. [46]

    Nguyen, K.; Tripathi, S.; Du, B.; Guha, T.; and Nguyen, T. Q. 2021. In defense of scene graphs for image captioning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 1407--1416

    2021

  39. [47]

    OpenAI. 2024. GPT-4 Technical Report. arXiv:2303.08774

    2024 arXiv

  40. [48]

    W.; Hallacy, C.; Ramesh, A.; Goh, G.; Agarwal, S.; Sastry, G.; Askell, A.; Mishkin, P.; Clark, J.; et al

    Radford, A.; Kim, J. W.; Hallacy, C.; Ramesh, A.; Goh, G.; Agarwal, S.; Sastry, G.; Askell, A.; Mishkin, P.; Clark, J.; et al. 2021. Learning transferable visual models from natural language supervision. In International conference on machine learning, 8748--8763. PMLR

    2021

  41. [49]

    Shi, J.; Zhong, Y.; Xu, N.; Li, Y.; and Xu, C. 2021. A simple baseline for weakly-supervised scene graph generation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 16393--16402

    2021

  42. [50]

    Suhail, M.; Mittal, A.; Siddiquie, B.; Broaddus, C.; Eledath, J.; Medioni, G.; and Sigal, L. 2021. Energy-based learning for scene graph generation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 13936--13945

    2021

  43. [51]

    Sung, Y.-L.; Cho, J.; and Bansal, M. 2022. Vl-adapter: Parameter-efficient transfer learning for vision-and-language tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 5227--5237

    2022

  44. [52]

    Tang, K.; Zhang, H.; Wu, B.; Luo, W.; and Liu, W. 2019. Learning to compose dynamic tree structures for visual contexts. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 6619--6628

    2019

  45. [53]

    Teney, D.; Liu, L.; and van Den Hengel, A. 2017. Graph-structured representations for visual question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, 1--9

    2017

  46. [54]

    Wang, H.; Yang, M.; Wei, K.; and Deng, C. 2023. Hierarchical prompt learning for compositional zero-shot recognition. In Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence, 1470--1478

    2023

  47. [55]

    Wang, Y.; Jiang, X.; Cheng, D.; Li, D.; and Zhao, C. 2024. Learning Hierarchical Prompt with Structured Linguistic Knowledge for Vision-Language Models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, 5749--5757

    2024

  48. [56]

    Wang, Z.; Lu, Y.; Li, Q.; Tao, X.; Guo, Y.; Gong, M.; and Liu, T. 2022 a . Cris: Clip-driven referring image segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 11686--11695

    2022

  49. [57]

    Wang, Z.; Wang, P.; Liu, T.; Lin, B.; Cao, Y.; Sui, Z.; and Wang, H. 2022 b . HPT: Hierarchy-aware prompt tuning for hierarchical text classification. arXiv preprint arXiv:2204.13413

    2022 arXiv

  50. [58]

    Wu, J.; Li, X.; Xu, S.; Yuan, H.; Ding, H.; Yang, Y.; Li, X.; Zhang, J.; Tong, Y.; Jiang, X.; et al. 2024. Towards open vocabulary learning: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence

    2024

  51. [59]

    Wu, X.; Zhu, F.; Zhao, R.; and Li, H. 2023. CORA: Adapting CLIP for Open-Vocabulary Detection with Region Prompting and Anchor Pre-Matching. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 7031--7040

    2023

  52. [60]

    Xian, Y.; Choudhury, S.; He, Y.; Schiele, B.; and Akata, Z. 2019. Semantic projection network for zero-and few-label semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 8256--8265

    2019

  53. [61]

    B.; and Fei-Fei, L

    Xu, D.; Zhu, Y.; Choy, C. B.; and Fei-Fei, L. 2017. Scene graph generation by iterative message passing. In Proceedings of the IEEE conference on computer vision and pattern recognition, 5410--5419

    2017

  54. [62]

    Yang, X.; Tang, K.; Zhang, H.; and Cai, J. 2019. Auto-encoding scene graphs for image captioning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 10685--10694

    2019

  55. [63]

    Yao, T.; Pan, Y.; Li, Y.; and Mei, T. 2018. Exploring visual relationship for image captioning. In Proceedings of the European conference on computer vision (ECCV), 684--699

    2018

  56. [64]

    Yu, Q.; Li, J.; Wu, Y.; Tang, S.; Ji, W.; and Zhuang, Y. 2023. Visually-Prompted Language Model for Fine-Grained Scene Graph Generation in an Open World. arXiv preprint arXiv:2303.13233

    2023 arXiv

  57. [65]

    Yuan, H.; Zhang, S.; Wang, X.; Albanie, S.; Pan, Y.; Feng, T.; Jiang, J.; Ni, D.; Zhang, Y.; and Zhao, D. 2023. RLIPv2: Fast Scaling of Relational Language-Image Pre-training. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 21649--21661

    2023

  58. [66]

    Yuksekgonul, M.; Bianchi, F.; Kalluri, P.; Jurafsky, D.; and Zou, J. 2022. When and why vision-language models behave like bags-of-words, and what to do about it? arXiv e-prints, arXiv--2210

    2022

  59. [67]

    D.; Hu, D

    Zareian, A.; Rosa, K. D.; Hu, D. H.; and Chang, S.-F. 2021. Open-vocabulary object detection using captions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 14393--14402

    2021

  60. [68]

    Zellers, R.; Yatskar, M.; Thomson, S.; and Choi, Y. 2018. Neural motifs: Scene graph parsing with global context. In Proceedings of the IEEE conference on computer vision and pattern recognition, 5831--5840

    2018

  61. [69]

    Zhang, C.; Stepputtis, S.; Campbell, J.; Sycara, K.; and Xie, Y. 2024. HiKER-SGG: Hierarchical Knowledge Enhanced Robust Scene Graph Generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 28233--28243

    2024

  62. [70]

    Zhang, C.; Yu, J.; Song, Y.; and Cai, W. 2021 a . Exploiting edge-oriented reasoning for 3d point-based scene graph analysis. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 9705--9715

    2021

  63. [71]

    Zhang, R.; Fang, R.; Zhang, W.; Gao, P.; Li, K.; Dai, J.; Qiao, Y.; and Li, H. 2021 b . Tip-adapter: Training-free clip-adapter for better vision-language modeling. arXiv preprint arXiv:2111.03930

    2021 arXiv

  64. [72]

    Zhang, Y.; Pan, Y.; Yao, T.; Huang, R.; Mei, T.; and Chen, C.-W. 2023. Learning To Generate Language-Supervised and Open-Vocabulary Scene Graph Using Pre-Trained Visual-Semantic Space. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2915--2924

    2023

  65. [73]

    Zheng, C.; Lyu, X.; Gao, L.; Dai, B.; and Song, J. 2023. Prototype-based embedding network for scene graph generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 22783--22792

    2023

  66. [74]

    ???? Learning to Generate Scene Graph from Natural Language Supervision

    Zhong, Y.; Shi, J.; Yang, J.; Xu, C.; and Li, Y. ???? Learning to Generate Scene Graph from Natural Language Supervision. In 2021 IEEE. In CVF International Conference on Computer Vision (ICCV), 1803--1814

    2021

  67. [75]

    H.; Zhou, L.; Dai, X.; Yuan, L.; Li, Y.; et al

    Zhong, Y.; Yang, J.; Zhang, P.; Li, C.; Codella, N.; Li, L. H.; Zhou, L.; Dai, X.; Yuan, L.; Li, Y.; et al. 2022. Regionclip: Region-based language-image pretraining. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 16793--16803

    2022

  68. [76]

    C.; and Liu, Z

    Zhou, K.; Yang, J.; Loy, C. C.; and Liu, Z. 2022. Learning to prompt for vision-language models. International Journal of Computer Vision, 130(9): 2337--2348

    2022

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.