{"id":"156a272a-cbbf-4d16-ad11-cff71e2dc72d","arxiv_id":"2412.19021","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A hierarchical prompt framework with entity clustering, LLM region descriptions, and VLM-based selection improves open-vocabulary scene graph generation on Visual Genome and Open Images v6.","lead":"This paper introduces RAHP, a way to make scene graph generation work for unseen relationships by adding richer text prompts generated from entity and region details. The method reports state-of-the-art results on two benchmarks, Visual Genome and Open Images v6.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The region-aware prompt mechanism is not directly validated for visual grounding; the reported gains could stem from an enlarged prompt vocabulary rather than from relation-specific region-text alignment.","rationale":"The reader's weakest assumption matches the most load-bearing concern I find: the region-aware prompt generation and dynamic selection are not validated for visual grounding, yet the central mechanism and SOTA claim depend on that grounding. The ablation evidence supports the framework's internal consistency but not the specific explanatory claim that region-level text descriptions improve alignment by capturing fine-grained visual interactions. Other issues noted by the reader (missing error bars, unfinished appendix placeholders, small and partly confounded baseline sets) are real but secondary; they affect confidence in the SOTA claim without undermining the controlled comparisons. A direct grounding test or a control experiment removing relation-specific text would settle whether the region-aware component is load-bearing. Because the reader already reached CONDITIONAL on this basis, and the proposed test could be added without changing the method, the verdict remains UNCHANGED.","tokens_in":19810,"tokens_out":2938,"duration_ms":32974,"concrete_test":"On the VG validation set, replace the LLM region prompts with a control set of k object-part prompts that contain no relation verbs (e.g., 'human hand', 'furniture seat'), keeping the same dynamic-selection procedure and k=3, and re-run PE-NET+RAHP. If novel mR@100 does not drop substantially relative to the real region prompts, the relation-specific content of the region prompts is not load-bearing. Additionally, measure the agreement between the top-k selected prompts and the ground-truth predicate using adjusted mutual information, controlling for subject and object categories; near-zero AMI would indicate that the selection is not relation-aware.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that RAHP 'consistently achieves state-of-the-art performance' rests on the assumption that LLM-generated region descriptions, produced without seeing images, yield CLIP text embeddings that align with the visual features of the relevant image regions, so that the top-k selection in Eq. 3 and the aggregation in Eq. 5 improve predicate classification. The paper's own ablation (Table 4) shows that region-aware prompts without dynamic selection hurt novel predicates (novel R@50 drops from 6.15 to 2.38 in row 3 vs. row 1), so the entire benefit of the region-aware component is attributed to dynamic selection. However, dynamic selection chooses prompts by CLIP similarity between the union-box feature U and region-text embeddings; this similarity may be driven by generic object or part recognition (e.g., 'hand', 'seat') rather than by the predicate-specific interaction. Because the evaluation uses Recall@K, adding more text prompts can inflate recall even if the prompts merely match generic visible parts, and the absence of any direct grounding measure leaves this confound unresolved. The paper states in Sec. 6 that limited diversity in LLM descriptions can hinder generalization, but it never verifies that the selected prompts are actually aligned with the visual relation. Thus the claimed mechanism of 'fine-grained visual interactions' is an unverified explanation for the observed improvements.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes RAHP, a prompt-construction framework for CLIP-based open-vocabulary scene graph generation (OV-SGG). It clusters entity categories into super entities, generates entity-aware prompts by combining super entities with predicates, and uses an LLM to produce region-aware part-level descriptions for each super-entity triplet. A VLM-guided dynamic selection step chooses the top-k region prompts per union-box visual feature, and predicate scores are obtained by a weighted average of entity-aware and region-aware similarities. Experiments on Visual Genome and Open Images v6 under OVR-SGG and OVD+R-SGG settings report consistent gains over fixed-prompt counterparts, with ablations decomposing the contributions of the three components. Code is released.","tokens_in":20105,"tokens_out":7787,"duration_ms":66246,"significance":"The paper addresses a real bottleneck in OV-SGG: fixed text prompts limit CLIP-based relation classification. The entity-clustering trick is a practical way to control the combinatorial explosion of triplets, and the idea of generating region-level descriptions with an LLM is timely. The controlled within-backbone comparisons (PE-NET+RAHP vs. PE-NET+p, SGTR†+RAHP vs. SGTR†+p) show credible gains on novel predicates, the ablation isolates components, and the held-out novel predicate evaluation is external to the prompt-generation process. If the mechanism is validated, RAHP would be a useful and transferable recipe for other VLM-based open-vocabulary tasks. However, the empirical support for \"consistently state-of-the-art\" is weakened by single-run results, partly confounded cross-backbone comparisons, and the absence of direct evidence that the region prompts selected by Eq. (3) are actually grounded in the visual interaction.","major_comments":[{"comment":"The abstract and Sec. 6 state that RAHP \"consistently achieves state-of-the-art performance\", but the cross-method comparisons in Tables 1 and 2 are not controlled. The headline improvement over PGSG (e.g., 21.56 points in total R@100 on OIV6) compares models with different backbones and detectors (ViT DETR* vs. R-101 DETR, R-101 Faster R-CNN) and different predicate splits. The controlled gains (e.g., PE-NET+RAHP vs. PE-NET+p) are credible, but the manuscript should either soften the SOTA claim or present like-for-like comparisons with published methods under identical backbones and splits.","section":"§5.2, Tables 1 and 2"},{"comment":"The central novelty claim is that region-aware prompts capture fine-grained visual interactions, but no experiment verifies that the prompts selected by Eq. (3) are visually grounded. Table 4 row 3 shows that adding region-aware prompts without dynamic selection drops novel R@50 from 6.15 to 2.38, so the entire benefit of the region-aware component is attributed to selection. Selection is based on CLIP similarity with the union box, which may be driven by generic part/object recognition rather than predicate-specific interaction. The paper should provide direct evidence—e.g., analysis of selected prompts per predicate, comparison with random or static prompt subsets, or a region-grounding metric—before attributing the gain to \"fine-grained visual interactions\".","section":"§4.4, Eq. (3) and Table 4"},{"comment":"All reported numbers are single-run point estimates with no error bars or significance tests. Some controlled margins are small—e.g., SGTR†+RAHP vs. SGTR†+p on VG: total R@50 39.92 vs. 39.48 and base R@50 41.29 vs. 40.78—so the \"consistently\" claim requires multiple seeds (at least 3) and reported mean ± std, especially for novel-predicate metrics.","section":"Tables 1–3"},{"comment":"The definition of the selection score is ambiguous. T^r_j is defined in Sec. 4.2 as an element of R^{C_p × N^r_j × d}, but Eq. (3) computes φ(U, T^r_j) with φ defined in Eq. (1) for 2D inputs, and writes the output as S^se_j ∈ R^{N × N^r_j}. It is unclear whether selection is performed separately per predicate or shared across all C_p predicates, and how Eq. (5) maps the selected 3D tensor to the 2D score S^r_j. Please clarify the tensor shapes and indexing; this is essential for reproducibility of the dynamic-selection mechanism.","section":"§4.4, Eqs. (3)–(5)"},{"comment":"In the fully open-vocabulary OVD+R-SGG setting, OvSGTR+RAHP improves novel predicate R@100 from 11.19 to 16.46 but decreases novel object R@50/R@100 from 14.37/17.44 to 12.45/15.38, and total R@100 increases only from 16.36 to 16.52. The paper mentions this trade-off but does not analyze it. Since the framework is proposed for open-vocabulary scene graph generation, the reported degradation on novel objects should be investigated or clearly qualified before claiming \"state-of-the-art generalization\" in this setting.","section":"§5.3, Table 3"}],"minor_comments":[{"comment":"The reference list contains entries unrelated to the paper (Alpher et al. 2002–2005; Anonymous 2024a,b) that appear to be leftover template text; they should be removed.","section":"References"},{"comment":"The sentence \"The total cost of generating region-aware prompts ... is [insert cost here]\" is an unfinished placeholder and must be completed or deleted.","section":"Appendix E.7"},{"comment":"The implementation details refer to \"PR-NET\" but the rest of the paper and Table 4 use \"PE-NET\"; please correct the typo.","section":"§5.1"},{"comment":"There are typos in the table headers: \"Realtion\" in Table 4 and \"CPT-4o-mini\" in Table 7 should be \"Relation\" and \"GPT-4o-mini\".","section":"Table 4 and Table 7"},{"comment":"The statistics \"20.32 unique objects and 7.58 unique relations\" are unclear; please specify whether these are per triplet, per predicate, or across the whole prompt set.","section":"Appendix C.1"}],"recommendation":"major_revision","confidential_remarks":"The core idea is promising and the controlled within-backbone gains are credible. The main risk is that the central claim is stronger than the experimental design supports, particularly the lack of direct validation of the region-prompt grounding mechanism and the single-run point estimates. The SOTA claim should be softened or made like-for-like, and the ambiguity in Eqs. (3)–(5) should be resolved. I would not reject the paper, but the revision needs to address these load-bearing points."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The thing to know about this paper is that the central mechanism is less secure than the SOTA claim makes it look. RAHP's region-aware prompts alone hurt novel-predicate recall in the ablation, and the entire gain comes from dynamic selection. That selection is based on CLIP similarity between the union-box feature and region-text embeddings, which could be matching generic object parts rather than the predicate-specific interaction. There is no direct grounding measure, so the paper's story about fine-grained visual interactions is currently an interpretation, not a demonstrated fact.\n\nThe genuinely new piece is the combination of entity clustering into super-entities, LLM-generated region descriptions, and VLM-guided top-k selection. That extends RECODE's subject-object-spatial prompts in a nontrivial way. The within-backbone controlled comparisons are the paper's strongest evidence: PE-NET+RAHP consistently beats PE-NET+p on both base and novel predicates, and SGTR+RAHP beats SGTR+p on OIV6. Table 3 shows meaningful novel-relation gains in the fully open OVD+R setting. The code is released, which helps reproducibility. The entity-aware prompt alone is also clean in the ablation and improves both base and novel recall.\n\nThe soft spots, in order of severity. First, the ablation in Table 4 is the load-bearing weakness: row 3 shows region-aware prompts without dynamic selection drop novel R@50 from 6.15 to 2.38, and row 4 shows they don't help even with entity-aware prompts. The jump to 9.25 only appears with dynamic selection. So the paper claims the region-aware component captures fine-grained visual interactions, but its own numbers say the region descriptions are noise unless filtered. And the filter uses a union-box feature that mostly encodes objects and parts, so it may simply be picking prompts that mention visible objects. Recall@K is vulnerable to that kind of inflation. Second, all results are single-run point estimates with no error bars; the SOTA claim relies on a small set of baselines, some with different backbones. Third, the paper is not camera-ready: the references still contain Alpher/frobnicate placeholders, and the appendix has an \"insert cost here\" string.\n\nWho should read this: people working on OV-SGG or on prompt construction for CLIP-style relation classifiers. It deserves a serious referee; the method is sensible and the gains look real. My review would ask for a diagnosis of why region prompts alone hurt, and for a direct test of whether the dynamic selection is relation-aware or just object-aware. I would also ask for error bars and a pass over the appendix.\n\nSend it to review, but with a clear expectation of revision.","headline":"RAHP is a plausible and well-controlled empirical extension of hierarchical prompting for open-vocabulary scene graph generation, but the paper's own ablation shows the region-aware prompts only work when paired with dynamic selection, and the claimed grounding mechanism is not directly validated.","tokens_in":20622,"tokens_out":2786,"would_cite":true,"duration_ms":29255,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that replacing fixed predicate prompts with hierarchical entity-aware and region-aware text prompts, filtered by a vision-language model, consistently improves open-vocabulary scene graph generation on Visual Genome and…","keywords":["open-vocabulary scene graph generation","visual relationship detection","vision-language models","hierarchical text prompts","entity clustering","region-aware prompt mining","dynamic prompt selection","zero-shot relation classification"],"falsifier":"If the top-k region prompts selected by vision-language matching are no more predictive of the true predicate than k prompts drawn at random from the same pool, then the dynamic selection mechanism is not contributing; that comparison is directly computable on Visual Genome using the paper's own pipeline.","tokens_in":19602,"feed_emoji":"🔗","tokens_out":8487,"duration_ms":71543,"temperature":0.7,"pith_summary":"The paper argues that open-vocabulary scene graph generation is held back by fixed, single-form text prompts for predicates, and that richer text representations can close the gap. To test this, the authors build RAHP, a framework that generates two levels of prompts: entity-aware prompts that combine predicates with clustered \"super entities,\" and region-aware prompts that use a large language model to describe the part-level visual interactions between subject and object. A vision-language model then filters the region prompts against the image content before scores from both levels are aggregated. The paper reports that this design consistently beats prior open-vocabulary systems on Visual Genome and Open Images v6, especially on novel predicates. If correct, the contribution is a lightweight text-side upgrade: any existing scene-graph predictor can gain open-vocabulary ability by swapping in richer, image-filtered prompts.","feed_headline":"Region-aware prompts boost unseen predicate recall","feed_subtitle":"Entity- and region-level prompts plus VLM filtering improve relation prediction on Visual Genome and Open Images.","key_machinery":"The carrying object is the hierarchical prompt stack plus a top-k filter. Entity clustering (WordNet grouping, VLM-embedding K-means, LLM naming) compresses the cubic triplet space into super-entity pairs so entity-aware prompts are feasible. A large language model then mines region-aware descriptions that name the part-level contact between subject and object. A vision-language model scores these region prompts against the union bounding-box crop, keeps the top k, and the final predicate logit is a weighted mixture of the entity-aware and region-aware similarities. The hyperparameters k and α control how many region prompts survive and how much they weigh.","core_discovery":"RAHP's central claim is that the quality of open-vocabulary relation prediction is determined less by the visual encoder than by the text representation it must match. The framework keeps the vision-language model's text encoder frozen and instead expands the text side: predicates are combined with entity clusters to form entity-aware prompts, and an LLM decomposes subject and object into parts to generate region-aware descriptions such as \"human buttocks are making contact with the seat of the furniture.\" During inference, the union region of the subject-object pair is used to rank the region prompts, keep the top k, and weight them into the final predicate score. On Visual Genome and Open Images v6, the paper reports consistent state-of-the-art results, with the largest gains on novel (unseen) predicates.","pith_inferences":["Because the text side is frozen and only prompts change, the framework could port to other open-vocabulary tasks such as object detection, segmentation, or human-object interaction, where part-level region descriptions can be mined without visual grounding.","The paper's own ablation hints that the region descriptions are the main source of novel-relation gains; a direct test would be to see whether paraphrased or ground-truth-grounded descriptions change performance, which would isolate whether the LLM's wording or its part decomposition carries the signal.","The weakest link is likely the diversity of LLM-generated descriptions; if one could measure visual grounding of each region prompt or add image-conditioned prompt generation, the top-k selection might become even more effective, and the method might extend beyond the datasets tested.","Entity clustering granularity is a latent trade-off: too few super entities wash out object identity, too many reintroduce the triplet-space explosion; this suggests an adaptive clustering scheme as a natural next step."],"forward_implications":["Any one-stage or two-stage scene graph model can adopt RAHP by replacing its fixed predicate classifier, as demonstrated with SGTR+, PE-NET, OvSGTR, and VS3.","Novel (unseen) predicate recall improves substantially; for example, PE-NET+RAHP raises novel mR@100 from 19.32 to 23.73 on Visual Genome PredCLS over a fixed-prompt baseline.","Filtering region prompts with the vision-language model is necessary: using all region prompts hurts novel predicate performance, while keeping only the top-1 hurts diversity.","The method transfers to fully open-vocabulary detection+relation settings, improving novel relation recall on OVD+R-SGG.","Entity clustering to 30 super entities on Visual Genome recovers nearly all the benefit of using all 1,125,000 full triplets."],"supporting_citations":[{"why":"Supplies the vision-language model whose frozen encoder defines the image-text matching space the framework builds on.","marker":"Radford et al. 2021"},{"why":"Introduces using LLM-generated descriptions as text prompts, the basis RAHP extends to the region level.","marker":"Menon and Vondrick 2022"},{"why":"RECODE, a zero-shot relation detection method that also exploits LLM composite cues; RAHP contrasts its regional perspective and adds dynamic selection.","marker":"Li et al. 2024b"},{"why":"OvSGTR, a baseline and task-defining method for open-vocabulary scene graph generation.","marker":"Chen et al. 2023"},{"why":"PE-NET, a two-stage SGG model used as a host for RAHP in the main experiments and ablations.","marker":"Zheng et al. 2023"},{"why":"SGTR+, a one-stage transformer SGG model used to validate RAHP under a different architecture.","marker":"Li, Zhang, and He 2024"},{"why":"Provides the entity-clustering strategy for reducing the triplet category space in hierarchical prompt generation.","marker":"Zhang et al. 2024"},{"why":"Visual Genome benchmark dataset used for the main open-vocabulary scene graph experiments.","marker":"Krishna et al. 2017"},{"why":"Open Images v6 benchmark dataset used for the second set of experiments.","marker":"Kuznetsova et al. 2020"}],"fun_headline_variants":["Hierarchical prompts beat fixed text for open-vocab scene graphs","Entity-aware prompts and VLM selection cut unseen relation errors","RAHP: Relation-aware hierarchical prompts for open-vocab SGG","Adaptive region-aware prompts beat fixed text for novel predicates"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole mechanism presumes that region-aware text descriptions written by an LLM without seeing an image land close, in the vision-language model's embedding space, to the visual features of the actual image region, so that top-k selection keeps helpful prompts rather than noise.","fun_headline_variants_meta":{"raw":{"variants":["Hierarchical prompts beat fixed text for open-vocab scene graphs","Entity-aware prompts and VLM selection cut unseen relation errors","RAHP: Relation-aware hierarchical prompts for open-vocab SGG","Adaptive region-aware prompts beat fixed text for novel predicates"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000606,"raw_usage":{"total_tokens":2815,"prompt_tokens":924,"completion_tokens":1891,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":540,"completion_tokens_details":{"reasoning_tokens":1819}},"tokens_in":540,"tokens_out":1891,"duration_ms":14287,"temperature":1.0,"reasoning_tokens":1819,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T00:58:25.761769+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"If the top-k region prompts selected by vision-language matching are no more predictive of the true predicate than k prompts drawn at random from the same pool, then the dynamic selection mechanism is not contributing; that comparison is directly computable on Visual Genome using the paper's own pipeline.","supporting_citations":[{"cited_title":"Expanding Scene Graph Boundaries: Fully Open-vocabulary Scene Graph Generation via Visual-Concept Alignment and Retention","cited_arxiv_id":"2311.10988","evidence_quote":"OvSGTR, a baseline and task-defining method for open-vocabulary scene graph generation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"PE-NET, a two-stage SGG model used as a host for RAHP in the main experiments and ablations."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"SGTR+, a one-stage transformer SGG model used to validate RAHP under a different architecture."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the entity-clustering strategy for reducing the triplet category space in hierarchical prompt generation."}],"review_version":1}