{"id":"d7692021-f3e7-44fe-88f2-c6da4e01d387","arxiv_id":"2501.09041","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"G2 generates a location-free scene graph from an image and feeds it, with confidence-based token weighting, into an LLM to produce visual commonsense answers and explanations.","lead":"A team trained an AI to first build a text description of the objects and relationships in a photo, then use that description to answer questions about the photo and explain its reasoning. The method improved scores on three visual question-answering benchmarks, though part of the gain may come from the stronger language model it uses.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The object-list input for VCR scene-graph generation is unspecified; Section 3.2 only defines Xo for VG training, yet Figure 2 requires objects like 'cup' and 'dining table' not present in the question.","rationale":"The reader's weakest assumption identifies exactly the same load-bearing concern: the source of the object list for VCR images is never specified. This is the most critical gap because the scene-graph generation stage cannot function without Xo, and the paper's own examples require objects beyond the question text. The internal ablation in Table 4 shows that scene graphs improve over the no-SG baseline with the same backbone, which supports the mechanism in principle, but that support only holds if the object list is legitimate and not oracle-derived. The missing specification is a reproducibility and possible-leakage concern, not a demonstrated falsehood, so the existing CONDITIONAL verdict remains appropriate. No code is released, and the paper does not state how VCR object lists are generated, which strengthens the need for a conditional acceptance pending clarification or release.","tokens_in":18816,"tokens_out":5124,"duration_ms":49612,"concrete_test":"Reimplement the described pipeline and run the location-free SGG stage on a random VCR validation image using Xo built solely from personN tokens appearing in the question (e.g., 'person1; person3; person6'). Then inspect the generated scene graph for objects such as 'cup', 'dining table', 'tie', and 'handbag'. If these objects never appear, the Figure 2 object list must have come from an unstated source, confirming the specification gap. As a follow-up, re-run the VCR experiments with Xo supplied by an off-the-shelf open-vocabulary detector (e.g., GroundingDINO) to quantify how much of the reported improvement depends on the missing object source.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.2 defines the scene-graph generator's text input Xo as 'subjects and objects from Q', but for VCR the only text Q is the question, e.g., 'what are person1, person3, and person6 doing?'. The example in Figure 2 and the case studies show object lists containing 'cup', 'dining table', 'tie', and 'handbag' — objects that never appear in that question. VCR provides no ground-truth object annotations, and the paper never describes an object detector, a CLIP-based naming step, or any other mechanism that produces these objects for VCR images. If Xo is limited to question entities, the generated scene graph cannot mention unmentioned objects, making the Figure 2 scene graph impossible. If Xo is obtained from an external detector or from the ground-truth answer/explanation, then the 'location-free' construction is not self-contained, and the comparison with baselines may leak privileged information. Because the entire claimed benefit of scene graphs in VCR rests on this object list, the paper's central pipeline is underspecified at its most load-bearing point.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes G2, a two-stage generative framework for Visual Commonsense Reasoning (VCR) under the VL-NLE setting. In the first stage, a location-free scene graph generator is trained on Visual Genome using CLIP patch features and Llama-3.2, with a text prompt that includes an object list. In the second stage, the generated scene graph triplets are fed, together with the image and question, into a second Llama-3.2 model that produces the answer and explanation; a CLIP-based confidence score for each triplet is used to weight the input tokens during training. The authors report experiments on VCR, VQA-X, and e-SNLI-VE, and claim that the scene-graph-enhanced pipeline outperforms prior generative VL-NLE baselines and that an automatic confidence-based selection mechanism is superior to threshold-based selection.","tokens_in":19080,"tokens_out":4356,"duration_ms":44218,"significance":"If the claims hold, the paper would provide a practical demonstration that LLM-generated, location-free scene graphs can improve generative visual commonsense answering and explanation, and the confidence-weighted token selection idea is a useful mechanism for injecting structured knowledge into a decoder-only model. The internal ablation in Table 4 and the visualizations in Figures 5-8 give some support for the central mechanism. However, the significance is limited by two issues: the provenance of the object-list input for VCR is unspecified, and the headline comparisons against prior work are confounded by backbone differences. The scene graph generation comparison against Pix2SG is also not controlled for the object-list input. These issues affect the strength of the main claim rather than only the presentation.","major_comments":[{"comment":"The provenance of the object list X_o used as input to the scene graph generator for VCR images is unspecified. The text defines X_o as 'subjects and objects from Q', but for VCR the only Q is the question, e.g., 'what are person1, person3, and person6 doing?', while Figure 2 and the case studies show object lists containing 'cup', 'dining table', 'tie', and 'handbag' that do not appear in the question. VCR provides no ground-truth object annotations, and the paper does not describe any detector, CLIP-based naming step, or other mechanism that produces these objects for VCR images. Because the generated scene graph is the only new signal introduced by G2, the reader cannot determine whether the improvement in Table 2 comes from the scene graph itself or from object information that is derived from an undocumented oracle or from the ground-truth answer/explanation. Please specify exactly how X_o is obtained at inference time for VCR, and provide an ablation that either removes the object-list input or obtains it from a described, non-oracle source.","section":"3.2, Figure 2"},{"comment":"The location-free scene graph generation comparison against Pix2SG is not apples-to-apples. Pix2SG is a location-free SGG method that does not receive an object list, whereas G2 is given X_o, a list of object names, as part of the text prompt. Since this object list fixes the node vocabulary of the generated scene graph, the large improvements at R@50 and R@100 (29.93 vs. 24.81 and 44.76 vs. 26.66) may reflect the provided object set rather than better relationship prediction. The paper should report a variant of G2 that does not receive the object list, or an equivalent setting for Pix2SG, before claiming superiority in location-free SGG.","section":"Table 3, Section 5.1"},{"comment":"The headline comparisons on VCR, VQA-X, and e-SNLI-VE are confounded by backbone and pretraining differences. G2 is initialized from Llama-3.2-1B, while the baselines e-UG, OFA-X, NLX-GPT, and UMAE use GPT-2 or OFA backbones. A newer and larger decoder can explain a substantial part of the gains in n-gram and BERTScore metrics, so the current Table 2 does not isolate the contribution of scene graphs. The 'G2 (w/o SG)' row in Table 4 is a useful start, but it should be included in the main comparison table, and the authors should add a same-backbone scene-graph-free baseline that reproduces the full G2 fusion and training setup, in order to support the claim that scene graphs are the source of the improvement.","section":"Table 2, Sections 4.2 and 4.4"}],"minor_comments":[{"comment":"The human evaluation is reported only for G2 (filtered and unfiltered), with no comparison to the baselines or to the ground-truth explanations, and no inter-annotator agreement measure, which makes the 63.1% 'well demonstrated' figure difficult to interpret.","section":"Figure 3 and Section 5.2"},{"comment":"The choice of 0.92 as the BERTScore threshold for filtering 'correct' answers is presented without justification or sensitivity analysis; please report unfiltered scores as well, or at least cite a precedent that uses the same threshold.","section":"Section 4.3"},{"comment":"Several entries are missing or unexplained, including the n-gram scores for UMAEVCR and some baseline scores on e-SNLI-VE; the paper should state explicitly which numbers are unavailable and why.","section":"Table 2"},{"comment":"There are numerous typos and inconsistencies: 'as shwon' in the Introduction, 'instancess' in Section 4.1, 'carring' and 'Selecction' in Figure 2, 'SSG' for SGG in Section 5.1, 'his generated results' in Section 5.2, and inconsistent 'GenGen' versus 'G2' labels in Figures 5-8.","section":"Throughout"},{"comment":"The caption of Table 1 says 'that contain goals and relationships'; this should read 'objects and relationships'.","section":"Table 1"}],"recommendation":"major_revision","confidential_remarks":"The central concern is the unspecified object-list input for VCR. During revision, the authors should be asked to provide a precise, reproducible description of how X_o is obtained for VCR images and to rule out the possibility that it is derived from the ground-truth answer or explanation. If the object list is indeed oracle-derived, the main comparison in Table 2 would be invalid and the paper would require rejection; if it comes from an external detector, that detector must be described and ablated. I also recommend asking for a same-backbone baseline in the main table, since the current backbone confound weakens the novelty claim."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"G2 is a plausible two-stage pipeline: a Llama-3.2 model trained on Visual Genome reads CLIP patch tokens plus a text list of object names and emits a location-free scene graph; then a second Llama pass, with CLIP-confidence-weighted attention over the scene-graph triplets, generates the VCR answer and explanation. What is genuinely new is the specific combination — location-free SGG from patch tokens as a feeder for generative VL-NLE, plus the confidence-weighting mechanism. The paper earns credit for the clean internal ablation: Table 4 shows the confidence-weighted selection beats both no-scene-graph and fixed thresholds, and the gains are in the right direction. The SGG results on VG are also reasonable evidence that the first stage learns something without bounding boxes.\n\nThe soft spots are real. The main one: Section 3.2 defines the scene-graph generator's object input Xo only for VG training, where object lists are part of the dataset. For VCR, the paper never says where the object list comes from. The figure and case studies show objects like 'cup', 'dining table', and 'tie' that are not in the question. VCR has no ground-truth object annotations, and no detector or CLIP naming step is described. If the object list comes from an external detector, the location-free claim is weakened and the comparison to baselines may leak privileged information; if it comes from the question only, the shown scene graphs are impossible. This is the most load-bearing assumption on the VCR side, and it is underspecified.\n\nTwo smaller issues. The comparison to Pix2SG in Table 3 is not apples-to-apples: G2 receives an object list in the prompt and Pix2SG does not, so the recall comparison mostly shows that giving away the objects helps. And the headline VCR gains over prior work are backbone-confounded — Llama-3.2 vs GPT-2/OFA — though the within-backbone ablation mitigates that. No code is released, so the numbers are not independently checkable.\n\nOverall: the central claim that scene graphs help holds within their own setup, but the pipeline is not self-contained as described. The paper deserves serious reviewer time — the idea is sensible, the ablation is honest, and the object-list gap is fixable with a clear description or a detector. I'd send it to review, with the expectation that the provenance question gets answered before acceptance. A reading group might enjoy debating whether the object list is the real source of the improvement.","headline":"A sensible two-stage scene-graph pipeline with a clean internal ablation, undermined by an underspecified object-list input for VCR that could be the real source of the gains.","tokens_in":19568,"tokens_out":2210,"would_cite":true,"duration_ms":22100,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that generating location-free scene graphs from image patches and object prompts, then weighting the graph triplets by CLIP confidence during LLM generation, yields more accurate visual commonsense answers and…","keywords":["visual commonsense reasoning","natural language explanation generation","location-free scene graph generation","scene graph selection","large language models","multimodal fusion","CLIP patch features","confidence-weighted attention"],"falsifier":"Run G2 on the VCR test set with the object prompt $X_o$ restricted to noun phrases from the question, and compare CIDEr and BERTScore with the paper's full-object-prompt results; if the margin over the no-scene-graph baseline vanishes, the improvement is carried by unstated external object knowledge rather than by generated scene-graph triplets.","tokens_in":18646,"feed_emoji":"🖼️","tokens_out":8097,"duration_ms":73636,"temperature":0.7,"pith_summary":"Visual commonsense reasoning goes wrong, this paper argues, when a model answers from memorized language patterns instead of from the objects and relations actually present in the image. It proposes G2, a two-stage generative method: first a large language model converts CLIP image patches plus a prompt that lists the objects into a location-free scene graph of triplets such as 'person sit in chair; cup on dining table; person hold handbag', then a second LLM stage reads those triplets alongside the question and image patches to produce an answer and an explanation. The paper reports that G2 beats prior generative vision-language explanation baselines on the VCR benchmark (for example, CIDEr rises from 47.3 to 57.7 and BERTScore from 81.9 to 91.1 on filtered generations) and that automatic, confidence-weighted triplet selection outperforms fixed-threshold filtering. A reader should care because the method targets a concrete failure mode — answers that ignore the actual scene — without needing bounding boxes or region annotations.","feed_headline":"Scene graphs sharpen AI visual commonsense answers","feed_subtitle":"A two-stage pipeline pulls object relationships from image patches and weights them by confidence for better answers and explanations.","key_machinery":"The load-bearing object is the location-free scene graph: a set of triplets $\\langle subject, relation, object\\rangle$ generated left-to-right as a text sequence, without bounding boxes, from CLIP patch tokens and a prompt that lists candidate objects. Training on Visual Genome teaches the generator to propose triplets; at inference the triplets are scored by CLIP image-text similarity, each triplet's tokens are multiplied by its confidence score before attention, and the weighted triplet text is prepended to the question as context for the answer-and-explanation generator. Cross-modal fusion between visual and text embeddings uses a single-head attention plus a gated combination, and the scene graph triplet weight $\\alpha_{ij}$ in equation (6) is the mechanism that lets the model down-weight unreliable triplets.","core_discovery":"On its own terms, the paper establishes that explicitly turning a scene into object-relation-object triplets before answering makes an LLM ground its reasoning in image content rather than in language priors. On the VCR benchmark, G2 improves filtered CIDEr from 47.3 to 57.7 and filtered BERTScore from 81.9 to 91.1 over the strongest earlier generative baselines, and human raters judged 63.1% of filtered explanations as well-justifying their answers. The scene graph is generated, not oracle-supplied: a Llama-3.2 model trained on Visual Genome emits triplets from CLIP patch tokens and an object prompt, and these triplets are then passed, with confidence weighting, into the answer-and-explanation generator.","pith_inferences":["A natural extension, not pursued in the paper, is to treat the confidence-weighting trick as the reusable idea: any structured text whose reliability varies token by token — retrieved facts, knowledge-base triples, OCR output — could be slotted into the same soft-weighting mechanism.","The paper's object-prompt ambiguity suggests a concrete check for readers: if an external detector supplies the object lists, then the fair comparison to prior work holds that detector fixed, and the 'location-free' claim reduces to relation prediction without boxes rather than recognition without boxes.","A testable successor would replace CLIP with an LLM-based triplet plausibility score and see whether the gains compound; the ablation in Table 4 only varies the threshold, not the scoring model.","The two-stage design also points to a production recipe: train relation extraction once on a relation-rich corpus, freeze it, and reuse it as a plug-in context provider for any downstream question-answering LLM."],"forward_implications":["If the central claim holds, generative vision-language explanation systems can be improved without bounding-box annotations or region proposals: patch-level CLIP features plus an LLM are enough to supply relational context.","Confidence-weighted token input provides a trainable, threshold-free way to filter noisy structured knowledge, suggesting that soft weighting can replace hard filtering whenever a pretrained scorer can rate generated facts.","The two-stage design means the scene graph generator can be trained once on a large relation-rich dataset and then reused across downstream VCR, VQA-X, and e-SNLI-VE tasks, which the paper reports with improved overall e-ViL scores.","Because the method constrains the model to attend to triplets, generated explanations should name concrete objects and relations rather than generic reasoning; the paper's human evaluation claims 63.1% of filtered explanations justify the answer well."],"supporting_citations":[{"why":"Supplies the Visual Genome scene-graph training data used to train the location-free scene graph generator.","marker":"[22]"},{"why":"Defines location-free scene graph generation and provides Pix2SG, the baseline G2 compares against on Visual Genome.","marker":"[31]"},{"why":"Provides the e-ViL benchmark, the filtered-evaluation protocol, and the e-UG baseline for VCR, VQA-X, and e-SNLI-VE.","marker":"[21]"},{"why":"Defines the VCR dataset and task that is the paper's main testbed.","marker":"[55]"},{"why":"Provides CLIP, the source of patch-sequence visual features and of the triplet confidence scores used by the selection mechanism.","marker":"[37]"},{"why":"Provides the Llama-3.2 backbone used for both scene graph generation and answer-and-explanation generation.","marker":"[16]"},{"why":"Supplies the NLX-GPT baseline and the converted generative VCR split used for training and evaluation.","marker":"[39]"},{"why":"Supplies the UMAE baseline, the strongest comparison in Table 2 on VCR.","marker":"[48]"}],"fun_headline_variants":["Scene graphs ground AI reasoning in image content","G2: Build scene graph, then answer with confidence","Object triplets lift visual commonsense scores","From patches to scene graphs: better VCR answers","Two-stage G2 turns images into grounded answers"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that a usable list of object names is available for every VCR image, yet the paper never specifies where that list comes from: VCR has no ground-truth object annotations, Section 3.2 only describes combining subjects and objects from the question, and the example object lists include names like 'cup' and 'dining table' that do not appear in the question.","fun_headline_variants_meta":{"raw":{"variants":["Scene graphs ground AI reasoning in image content","G2: Build scene graph, then answer with confidence","Object triplets lift visual commonsense scores","From patches to scene graphs: better VCR answers","Two-stage G2 turns images into grounded answers"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000146,"raw_usage":{"total_tokens":1138,"prompt_tokens":854,"completion_tokens":284,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":470,"completion_tokens_details":{"reasoning_tokens":212}},"tokens_in":470,"tokens_out":284,"duration_ms":3298,"temperature":1.0,"reasoning_tokens":212,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T20:23:10.883396+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run G2 on the VCR test set with the object prompt $X_o$ restricted to noun phrases from the question, and compare CIDEr and BERTScore with the paper's full-object-prompt results; if the margin over the no-scene-graph baseline vanishes, the improvement is carried by unstated external object knowledge rather than by generated scene-graph triplets.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines location-free scene graph generation and provides Pix2SG, the baseline G2 compares against on Visual Genome."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the NLX-GPT baseline and the converted generative VCR split used for training and evaluation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the UMAE baseline, the strongest comparison in Table 2 on VCR."}],"review_version":1}