{"id":"671790b9-1a16-452f-8ef3-73f2924da883","arxiv_id":"2506.08189","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"A zero-shot VLM pipeline can generate scene graphs with unseen objects and relations, and a new open-world evaluation setting exposes how much capacity remains untapped.","lead":"The authors build a training-free pipeline that asks vision-language models to list objects, detect them, refine pairs, and predict relationships, producing scene graphs without any fine-tuning. They add a new fully open-world benchmark where both objects and relationships are unseen, and report small but nonzero recall on VG, Open Images, and PSG datasets.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"No image-blind control: OW recall may reflect text-only semantic priors from the image-agnostic pair-refinement stage (Eq. 1, §3.3.4), so the 'vision-language' central claim is not yet isolated from language-only reasoning.","rationale":"OwSGG is a reasonable training-free framework, and the proposed OW benchmark is a useful contribution. The reader's weakest assumption, that the image-agnostic semantic score in Eq. 1 caps recall, is real but not by itself fatal: non-zero OW recall already survives that filter. The more load-bearing issue is attribution. Because the same text-only semantic score and the object category names are available before pixels are used, non-zero OW recall could in principle be produced by a language-only model. A gray-image control would settle whether the 'vision' part of the central claim is actually exercised. The paper's limitation section is honest about pre-trained component error and context-length constraints, but it does not include this control. Additionally, no code is released, and hyperparameters such as alpha and beta appear to be selected using the reported ablations, so the quantitative evidence is conditional. None of this changes the reader's CONDITIONAL verdict; it sharpens the condition that the authors should isolate the visual contribution of the pipeline before the vision-language interpretation of the central claim is taken at face value.","tokens_in":24056,"tokens_out":9664,"duration_ms":125352,"concrete_test":"Use the already-detected objects and boxes from the original VG150 OW images, then re-run the semantic-refinement and relation-generation VLM stages with a uniform gray image in place of the original photograph, holding object names, bounding boxes, depth maps, prompts, and all hyperparameters fixed. Compare OW R@50/R@100 against the image-conditioned numbers in Table 3. If gray-image recall is within roughly 20% relative of the image-conditioned recall, the non-zero OW result is attributable to language/geometry priors rather than visual understanding; if recall collapses to near zero, the visual input is load-bearing and the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that pretrained VLMs support open-world SGG without task-level training, and the key evidence is non-zero OW recall, e.g., Qwen2-VL-72B R@50=1.61 / R@100=2.41 on VG150 (Table 3). The pipeline, however, contains a text-only bottleneck before any image-conditioned reasoning: §3.3.4 defines P^S_ij = VLM(o_i, o_j) using category labels alone, assigns that score to every instance of the pair, and uses it in Eq. 4 to select top-k pairs for relation prediction. Because VLM pretraining encodes strong language priors (e.g., person-horse -> riding, book-bookcase -> on), a correct OW triplet can in principle be produced from object names and box geometry without the model ever consulting pixel content. No control run removes or degrades the image input, so the non-zero OW numbers do not currently establish that visual understanding, rather than text/geometry priors, drives the result. This does not refute the 'zero additional learning' claim, but it undercuts the specifically vision-language interpretation of the central claim. The paper's own limitation section acknowledges component error and context-length constraints but does not include such a control, and no code is released to check the attribution independently.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes OwSGG, a training-free pipeline for open-world scene graph generation that uses pretrained vision-language models (VLMs) as the sole relational reasoner. The pipeline consists of VLM-based entity generation, SimCSE-based entity mapping to dataset categories, Grounding DINO detection, a pair-refinement stage that fuses an image-agnostic semantic compatibility score with a geometric distance score, and a final VLM prompt that predicts relations for the retained top-k pairs. The authors define a taxonomy of open-world settings (Close Vocabulary, Zero-Shot, OVR, OVD, OVD+R, and strictly Open World) and report recall/mRecall on VG150, OIV6, and PSG for LLaVA-Next-7B, Qwen2-VL-7B, and Qwen2-VL-72B. The main empirical claim is that VLMs produce non-zero open-world recall without any task-specific training, with the strongest OW numbers on VG150 (e.g., Qwen2-VL-72B R@50=1.61, R@100=2.41 in Table 3).","tokens_in":24327,"tokens_out":4084,"duration_ms":50900,"significance":"If the central claim holds, the paper makes a useful contribution: it provides a reproducible-style evaluation protocol for training-free open-world SGG, a lightweight pair-refinement strategy, and evidence that large VLMs contain enough relational knowledge to generate non-trivial scene graphs over novel object and relation vocabularies. The paper tests three VLM backbones on three standard datasets and includes ablations of the semantic and geometric refinement branches, which is a strength. The reported non-zero recall in the strictly open-world setting is a genuinely interesting finding. However, the attribution of that recall to visual understanding, rather than to language-only priors or geometry, is not yet isolated, and the evaluation protocol has several gaps concerning hyperparameter selection, output parsing, and baseline generation. These issues are fixable but must be addressed before the central claim is fully supported.","major_comments":[{"comment":"The paper does not include an image-blind control, so the non-zero OW recall in Table 3 cannot be attributed to visual understanding. The semantic pair-refinement score in Eq. (1), P^S_ij = VLM(o_i, o_j), is computed from category labels alone and assigned to every instance of the pair, then fused in Eq. (4) to select the top-k pairs before any image-conditioned relation prediction. Because VLM pretraining encodes strong language priors about which object categories typically interact, a correct OW triplet could in principle be produced from object names and box geometry without the model ever seeing pixel content. The authors should run a control that removes or degrades the image input (e.g., blank image, no image, or a text-only variant of the relation-generation step) and report whether the OW recall persists. Without such a control, the 'vision-language' interpretation of the central claim is not isolated.","section":"§3.3.4, Eq. (1), and Table 3"},{"comment":"The hyperparameters appear to be selected on the same data used for reporting, and no error bars or validation splits are provided. The final values (alpha=0.25, beta=10 for 72B and beta=16 for 7B models, top_k=25, tau=0.2, delta=0.05, k=2, lambda1=1.0, lambda2=1.5, distance threshold 0.5) are fixed in the supplementary, while Fig. 2 ablates alpha and beta with no mention of a held-out split. Since the reported R@K numbers are the main evidence for the central claim, the authors should either commit to a validation-based selection procedure or provide a sensitivity analysis showing that the OW results are robust across a range of hyperparameters. The lack of error bars is also a concern for a method that involves stochastic VLM decoding, even at temperature 0.1.","section":"§4.4 and Supplementary B.5"},{"comment":"The only non-zero OW baseline in Table 3 is OvSGTR with a dagger indicating 'results were generated for this work,' but the paper does not describe how these numbers were produced, what split filtering was applied, or whether the same OW triplet definition was used. Without this protocol description, the comparison cannot be verified or reproduced. Additionally, the corresponding table row may have been generated with different code than the original OvSGTR release, so the authors should either provide the exact protocol and code or clearly state that these numbers are preliminary. This is load-bearing because Table 3 is the only direct evidence that the proposed open-world setting is distinguishable from the closed-vocabulary baselines.","section":"Table 3 and §4.3"},{"comment":"The paper never specifies how free-form VLM relations (e.g., 'the woman is sitting on the chair') are mapped to the fixed predicate vocabularies of VG150, PSG, and OIV6, nor how multiple generated sentences per pair are converted into the confidence scores that underlie R@K and mR@K. This mapping is essential for reproducing any of the reported recall numbers and for understanding whether the evaluation is lenient (e.g., accepting synonym paraphrases) or strict (exact string match). The authors should describe the parsing, normalization, and matching procedure, and ideally release the mapping code. Without this, the numerical results in Tables 1-4 are not independently checkable.","section":"§3.3.5 and §4 (Evaluation Metrics)"}],"minor_comments":[{"comment":"The PredCls OVR rows for OwSGG on VG contain three comma-separated values per cell (e.g., '0.75 / 1.36 / 1.5' and '0.44 / 1.2 / 2.12'), while the table header specifies only mR@50/100 and R@50/100. This is likely a transcription error and should be corrected to two values per column.","section":"Table 2"},{"comment":"The row label 'HEIRCOM' in the OIV6 PredCls block should be 'VCTree+HIERCOM' for consistency with the VG block; also, the CaCao row in the VG PredCls block reports only three values under Close Vocabulary, leaving the reader unsure whether mR@20 is missing or intentionally blank.","section":"Table 1"},{"comment":"The caption says 'F1 scores across different (a) α and (b) β values for the Qwen-72B model,' but the figure legends include llava-next, qwen2_vl, and qwen2_72vl. The caption should be corrected to describe all three models.","section":"Figure 2"},{"comment":"There is a typo 'V ocabualry' in the section title, and the entity mapping ablation text says 'SBERT slightly outperforms SimCSE for Qwen72 on PSG' while the figure caption claims 'SimCSE yields up to a 5% recall boost on PSG and OI'; these statements are not obviously consistent and should be reconciled.","section":"Section 2 and Supplementary B.3.1"},{"comment":"References [47] and [48] are duplicates of the same paper; one should be removed and all citations updated.","section":"References"},{"comment":"No code or public evaluation scripts are released, which is particularly important for a paper whose main deliverable is a benchmark protocol and a set of baselines; releasing the parser, split definitions, and hyperparameter configuration would substantially strengthen reproducibility.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The paper's contribution is more of an evaluation protocol and empirical study than a new learning method, which is fine for the venue. The main risk is that the central 'vision-language' claim is not yet isolated from language-only priors; however, this is fixable with a control experiment, so I recommend major revision rather than rejection. I would also ask the editor to ensure that the authors provide the missing relation-parsing details and clarify the provenance of the daggered OvSGTR baseline, as these directly affect the credibility of the numerical comparisons."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The genuinely new thing here is the fully open-world setting: both object and relation classes are novel at test time, and the paper provides the first training-free baseline for it. That formalization plus the three-dataset evaluation (VG, OIV6, PSG) is a real service to the SGG community. The pipeline itself is a sensible composition of off-the-shelf parts — VLM entity enumeration, SimCSE mapping, Grounding DINO detection, depth-based geometric filtering, and a final VLM relation prompt. The ablations on alpha, beta, and the semantic/depth filter combination give useful information, and the paper is honest that it does not beat closed-vocabulary baselines in the closed setting.\n\nI agree with the main thrust of your take, and I'd flag several soft spots in proportion. First, no code is released, which matters more here than usual because the OW numbers are the headline and there is no independent baseline to check them against. Second, alpha, beta, top_k, tau, delta, and k are all fixed on the evaluation datasets, so the reported recall values are partially selected. Third, there are no error bars, and some cells in Table 1 look mis-transcribed — for example, the identical mR values for IMP and MOTIFS, and a non-monotonic CaCao row. None of these are fatal, but they all push in the same direction: treat the exact numbers as provisional.\n\nThe stress-test concern about the image-blind semantic filter is legitimate and worth naming. Eq. 1 scores object pairs from category names alone, before any image-conditioned reasoning, and that score decides which pairs survive to the final VLM. A correct OW triplet can therefore be produced from language priors plus geometry, without the model ever consulting pixel content. The final relation generation is image-conditioned, so the claim is not a pure language-only result, but the paper does not include a control that removes or degrades the image input. That would be a cheap and convincing experiment, and its absence undercuts the specifically vision-language interpretation of the central claim.\n\nWho is this for? Anyone working on open-vocabulary or open-world scene graph generation, especially people looking for a baseline to beat in the OW setting. It deserves a serious referee — the benchmark contribution is real, and the pipeline is a reasonable starting point. For the revision, I would ask for code, error bars or at least a stability analysis, a clear statement of how hyperparameters were chosen, and the image-blind control. With those it would be a solid reference point.","headline":"Useful first benchmark for fully open-world SGG with a training-free VLM pipeline, but the empirical claims need tightening before the numbers are taken at face value.","tokens_in":24975,"tokens_out":1854,"would_cite":true,"duration_ms":25838,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Pretrained VLMs can generate scene graphs over unseen objects and relations without any training, and this paper supplies a standard protocol for measuring that open-world ability.","keywords":["scene graph generation","open-world","zero-shot","vision-language models","training-free","pair refinement","open vocabulary"],"falsifier":"Take an image whose ground-truth scene graph contains a pair that is visually interacting but semantically unusual (for example, a squirrel sitting on a drone, or a person holding an octopus): the pair-refinement module must rank that pair in its top-$k$ using only the names 'squirrel/drone' or 'person/octopus', and the final recalled triplet must appear in the output. If such visually obvious but text-unusual pairs are consistently pruned, the claim that VLMs provide open-world relational understanding would be shown to be largely a spatial-semantic prior rather than a true image-conditioned reasoning ability.","tokens_in":23831,"feed_emoji":"🕸️","tokens_out":4225,"duration_ms":39907,"temperature":0.7,"pith_summary":"This paper asks whether an off-the-shelf vision-language model can produce a scene graph—a structured list of objects in an image and the relationships between them—when the object categories and relationship types were never seen during any task-specific training. The authors claim yes: their training-free pipeline, called OwSGG, extracts candidate objects by prompting a VLM, maps those names onto dataset vocabularies with a contrastive text encoder, localizes them with an open-vocabulary detector, prunes implausible pairs using a mix of semantic and depth-based scores, and then asks the VLM to name the relationship for each surviving pair. They also formalize an Open-World evaluation protocol that splits the test vocabulary into closed, zero-shot, open-vocabulary, and fully open-world settings, and they report non-zero recall on Visual Genome, Open Images V6, and PSG. A sympathetic reader would care because success would mean relational visual understanding is already latent in pretrained multimodal models, so scene-graph generation could be performed without dataset-specific fine-tuning, and the new protocol would let future systems be compared on that capability.","feed_headline":"VLMs can build scene graphs with zero training, new protocol shows","feed_subtitle":"A model-agnostic pipeline recalls novel object-and-relation triplets on VG, Open Images, and PSG, and defines the open-world benchmark.","key_machinery":"The load-bearing mechanism is the pair-refinement module, which fuses a VLM semantic score $P^S_{ij} = \\mathrm{VLM}(o_i, o_j)$ with a geometric score $P^G_{ij} = \\sigma(-\\beta(d_{ij}-\\tau))$ computed from 2D bounding-box centers and monocular depth estimates, then keeps the top-$k$ pairs under the weighted log combination $P^{\\text{combined}}_{ij} = \\alpha \\log P^S_{ij} + (1-\\alpha)\\log P^G_{ij}$. This is what turns the combinatorially large set of possible object pairs into a short, VLM-processable candidate list before the relation-prediction step; the semantic branch is what supplies open-world generalization, since it judges whether two category names plausibly interact without needing to have seen that exact pair, while the depth branch supplies spatial plausibility.","core_discovery":"The central discovery is that pretrained VLMs, guided by a fixed prompting and filtering pipeline and never fine-tuned on scene-graph data, can recover ground-truth relational triplets even when both the object names and the predicate names are novel relative to any training set; the paper's Table 3 reports R@50/R@100 of 1.92/2.56 for LLaVA-next and 1.61/2.41 for Qwen2-72B in the strictly Open-World setting on Visual Genome. The paper reads this as evidence that the semantic priors in VLMs are sufficient for compositional relational reasoning, not just for recognizing isolated objects. The framework is claimed to be model-agnostic in that the same pipeline was run with three different VLM backbones and produced the same qualitative pattern of results.","pith_inferences":["A testable extension the paper leaves implicit: the semantic compatibility score $\\mathrm{VLM}(o_i, o_j)$ could be replaced by a contrastive text-encoder score (e.g., the same SimCSE embeddings used in entity mapping), and comparing those two variants would isolate whether the VLM's generative semantic judgment adds anything beyond embedding similarity for pair recall.","A second extension: because the paper's pipeline is fully training-free, the same protocol could be run with the VLM replaced by a much larger or vision-centric model, or with the detector replaced by a different open-vocabulary detector, to measure how much of the open-world recall ceiling is set by each module rather than by the VLM itself.","The paper reports non-zero OW recall but does not analyze where the recalled triplets come from; a reasonable inference is that they are dominated by spatial relations (e.g., 'on', 'next to', 'behind') whose names co-occur with the novel object names in the VLM's training corpus, so the headline open-world numbers may be measuring commonsense spatial priors rather than truly novel functional relat","The authors' choice to evaluate only on test data with no training could be re-purposed as a data-annotation tool: a deployed OwSGG pipeline could pre-label candidate triplets for new object vocabularies, with a human only correcting the filtered pairs, which would lower the cost of building future SGG datasets."],"forward_implications":["If the claim holds, scene graph generation becomes a deployment-time capability of any sufficiently capable VLM plus a detector and a text encoder; no SGG-specific training runs are needed to add a new domain's objects or predicates.","The new Open-World split (novel objects and novel relations simultaneously) gives later methods a concrete, standardized baseline to beat, which the paper argues was previously missing.","Because the pipeline is model-agnostic, the same prompts and refinement code can be re-run as stronger VLMs appear, making performance on open-world SGG a direct comparison of the VLM's latent relational knowledge rather than of trained SGG architectures.","The paper's results in OVR on PSG (R@50/100 of 10.42/13.54 for Qwen2-72B, above the PGSG baseline) suggest that on some open-world axes a training-free VLM pipeline can match or exceed trained open-vocabulary methods, which would redirect effort toward better prompting and pair selection rather than dataset-specific supervision.","The qualitative ablation results (depth-only refinement winning in closed vocabulary, semantic-plus-depth winning in open settings) imply that as the evaluation domain becomes less familiar, the VLM's name-based semantic prior contributes more than spatial proximity."],"supporting_citations":[{"why":"Visual Genome is one of the three evaluation datasets and supplies the VG150 closed vocabulary (150 objects, 50 relations) used for the closed, zero-shot, and open-world splits.","marker":"[16]"},{"why":"Open Images V6 is the second evaluation dataset (601 objects, 30 relations) and provides the OIV6 splits used in PredCls and SGDet experiments.","marker":"[17]"},{"why":"Panoptic Scene Graph (PSG) is the third evaluation dataset (133 objects, 56 relations) and supplies the validation split used for the PSG results and ablations.","marker":"[40]"},{"why":"Grounding DINO is the open-vocabulary object detector that localizes each mapped entity name into bounding boxes, effectively filtering out entities not actually present in the image.","marker":"[26]"},{"why":"SimCSE is the contrastive sentence encoder used in the entity-mapping module to align free-form VLM entity names to the dataset's canonical object vocabulary.","marker":"[7]"},{"why":"The geometric distance formulation (2D center distance plus depth difference with weights and a threshold) is adopted from this prior work and converted into a soft sigmoid score in this paper.","marker":"[6]"},{"why":"Depth Anything V2 is the monocular depth estimator that produces the depth map used in the geometric pair-refinement branch.","marker":"[41]"},{"why":"LLaVA-Next is one of the three VLM backbones on which the OwSGG pipeline is evaluated.","marker":"[24]"},{"why":"Qwen2-VL (7B and 72B) is the other VLM backbone family and supplies the strongest reported results in several open-world settings.","marker":"[37]"},{"why":"RAHP is one of the open-vocabulary SGG baselines compared in the OVR and OvD+R/Ow tables, and it is also cited as an example of expensive O(n^2) pair refinement that OwSGG aims to make lightweight.","marker":"[27]"}],"fun_headline_variants":["Zero-training scene graph generation from VLMs now possible","VLMs generate open-world scene graphs with zero fine-tuning","No training needed: VLMs build scene graphs for unseen objects","Training-free open-world SGG: VLM priors suffice","VLMs recall novel relational triplets with zero SGG training"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The pipeline's ceiling is set by a text-only judgment: the VLM is asked, without seeing the image, how likely two object category names are to interact, and any pair that gets a low score is discarded before the image is ever consulted, so visually present but semantically unusual interactions are lost before the final relation prediction step.","fun_headline_variants_meta":{"raw":{"variants":["Zero-training scene graph generation from VLMs now possible","VLMs generate open-world scene graphs with zero fine-tuning","No training needed: VLMs build scene graphs for unseen objects","Training-free open-world SGG: VLM priors suffice","VLMs recall novel relational triplets with zero SGG training"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000478,"raw_usage":{"total_tokens":2351,"prompt_tokens":909,"completion_tokens":1442,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":525,"completion_tokens_details":{"reasoning_tokens":1359}},"tokens_in":525,"tokens_out":1442,"duration_ms":12832,"temperature":1.0,"reasoning_tokens":1359,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T05:17:10.356667+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take an image whose ground-truth scene graph contains a pair that is visually interacting but semantically unusual (for example, a squirrel sitting on a drone, or a person holding an octopus): the pair-refinement module must rank that pair in its top-$k$ using only the names 'squirrel/drone' or 'person/octopus', and the final recalled triplet must appear in the output. If such visually obvious but text-unusual pairs are consistently pruned, the claim that VLMs provide open-world relational understanding would be shown to be largely a spatial-semantic prior rather than a true image-conditioned reasoning ability.","supporting_citations":[{"cited_title":"Visual genome: Connecting language and vision using crowdsourced dense image annotations.International journal of computer vision, 123:32–73, 2017","cited_arxiv_id":null,"evidence_quote":"Visual Genome is one of the three evaluation datasets and supplies the VG150 closed vocabulary (150 objects, 50 relations) used for the closed, zero-shot, and open-world splits."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Open Images V6 is the second evaluation dataset (601 objects, 30 relations) and provides the OIV6 splits used in PredCls and SGDet experiments."},{"cited_title":"Panoptic scene graph gen- eration","cited_arxiv_id":null,"evidence_quote":"Panoptic Scene Graph (PSG) is the third evaluation dataset (133 objects, 56 relations) and supplies the validation split used for the PSG results and ablations."},{"cited_title":"Depth anything v2.Advances in Neural Information Processing Systems, 37: 21875–21911, 2024","cited_arxiv_id":null,"evidence_quote":"Depth Anything V2 is the monocular depth estimator that produces the depth map used in the geometric pair-refinement branch."},{"cited_title":"Relation-aware hierarchical prompt for open-vocabulary scene graph generation","cited_arxiv_id":null,"evidence_quote":"RAHP is one of the open-vocabulary SGG baselines compared in the OVR and OvD+R/Ow tables, and it is also cited as an example of expensive O(n^2) pair refinement that OwSGG aims to make lightweight."}],"review_version":1}