{"id":"57c5ea3d-1912-47f9-8dae-3d6a672af4b2","arxiv_id":"2505.09118","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"An interaction-augmented scene graph pipeline with chain-of-thought graph construction and reward-based tuning improves VLM reasoning on several benchmarks.","lead":"This paper adds an interaction-aware scene graph, built by a language model, on top of vision-language models so they can reason about actions such as catching or throwing. The authors report stronger scores on several interaction-heavy benchmarks, but the method hides key implementation details, the code is not available, and the training data comes from the model itself.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Eq. 15's reward function is unspecified and omits answer correctness, so the IRR component of the central claim cannot be verified or reproduced.","rationale":"The reader's verdict is CONDITIONAL and the rationale already mentions \"undefined reward components,\" but the reader's stated weakest assumption is the accuracy of ICoT-generated graphs. My stress-test identifies a more immediate and specific gap: even if the graphs were perfectly accurate, the reward driving the IRR phase is not specified and, as written, lacks any answer-correctness term. This directly affects the strongest claim, which uses the ISGR(SFT+IRR) model. The concern is not a disagreement with the reader's list of weaknesses; it is a sharper version of one item on that list. I considered two other candidate concerns: (1) potential training/evaluation contamination, since the 300K interaction-augmented data could overlap with benchmarks like MMT-Bench or SEEDBench, and (2) the absence of the most relevant baselines for the state-of-the-art claim. Both are real, but the reward gap is the least secure element because it affects the mechanism itself and blocks any independent reproduction of the headline numbers. A concrete test that defines and validates the reward would settle the issue; if the reward correlates with correctness and the reported gains reproduce, the concern is resolved. If not, the IRR contribution should be downgraded. This does not change the verdict from CONDITIONAL, so I recommend UNCHANGED.","tokens_in":17531,"tokens_out":11939,"duration_ms":121255,"concrete_test":"Request the exact reward implementation (code and prompt definitions for F_focus, F_disamb, F_rele) and run one sanity check: take the K sampled responses for 1000 held-out (image, question) pairs used in IRR, compute R(y_k) per Eq. 15, and measure the rank correlation (Spearman or Kendall) between R and human/benchmark answer correctness (A_gt). Also re-run ISGR(SFT+IRR) with an explicit answer-accuracy reward term (e.g., exact match or LLM-judge correctness) and compare the results to Table 3. If R does not correlate with correctness, or if the gains disappear under a correctness-based reward, then the IRR component of the central claim is unsupported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing gap is the undefined reward in Section 3.3.2. The paper defines L_IRR = E_{(I,Q,A)~D}[R(A_pred, A_gt)], but Eq. 15 gives R(y_k) = λ1·F_focus(y_k) + λ2·F_disamb(y_k) − λ3·F_rele(y_k), and F_focus, F_disamb, and F_rele are never defined anywhere. None of the three terms takes the ground-truth answer A_gt as input, even though the surrounding prose claims R evaluates \"the quality of the interaction graph and the correctness of the final answer.\" Without these definitions, the reader cannot know what signal GRPO is optimizing, and the reported gains attributed to IRR (+7.9% SEEDBench, +7.8% MMT-Bench, +7.1% RealWorldQA) are not reproducible from the paper. Table 3 further shows average differences of less than 1 point between the full reward and each single-component variant, so without error bars the IRR contribution may be within run-to-run noise. If F_* are LLM-as-judge or heuristic scores, their alignment with benchmark correctness must be demonstrated before the central claim is accepted.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Interaction-augmented Scene Graph Reasoning (ISGR), a framework aimed at improving vision-language models' reasoning about object interactions. The framework has three components: (i) a summarize-and-align graph constructor that builds spatial, abstract, and interaction graphs via an Interactional Chain-of-Thoughts (ICoT) process; (ii) generation of interaction-focused instruction-query pairs from these graphs for supervised fine-tuning; and (iii) Long-term Memory Reinforcement (LTMR) that combines SFT with a GRPO-based Interaction Reasoning Reinforcement (IRR) step using a composite reward. The authors evaluate ISGR variants on general VQA, spatial, and scene-reasoning benchmarks, reporting consistent gains over LLaVA-v1.5 and several baselines, with particularly large improvements on RealWorldQA, MMT-Bench, and SEEDBench. The paper includes ablations of query types, reward components, and spatial grounding, along with prompts, dataset statistics, and an algorithm pseudocode.","tokens_in":17755,"tokens_out":4897,"duration_ms":46611,"significance":"If the reported results hold, ISGR would be a useful contribution to structured interaction reasoning in VLMs, moving beyond spatial-only scene graphs and demonstrating that instruction tuning on interaction-augmented graphs can improve a range of multimodal benchmarks. The framework is modular, and the SFT data-construction pipeline is described in enough detail (prompts, dataset statistics, and ablation of query types) for others to reproduce the SFT component. However, the central IRR component is under-specified to the point of non-reproducibility, the closest scene-graph baselines are not compared, and the reported gains carry no uncertainty estimates. The paper therefore provides a promising but incompletely verified method; the SFT results and ablations are valuable, but the IRR claims need substantial additional specification and validation.","major_comments":[{"comment":"The reward function used in IRR is not defined. Eq. (15) gives R(y_k) = λ1·F_focus(y_k) + λ2·F_disamb(y_k) − λ3·F_rele(y_k), but the functions F_focus, F_disamb, and F_rele are never defined anywhere in the paper. Moreover, none of the three terms takes the ground-truth answer A_gt as input, even though Eq. (14) and the surrounding prose state that R evaluates both graph quality and answer correctness. As a result, L_IRR and the reported IRR gains (e.g., +7.9% on SEEDBench, +7.8% on MMT-Bench, +7.1% on RealWorldQA) cannot be reproduced or verified from the manuscript. Please provide exact operational definitions of F_focus, F_disamb, and F_rele (or cite a specific implementation), and clarify how the reward is used within the GRPO objective (e.g., advantage normalization, KL penalty, and number of sampled responses K).","section":"Section 3.3.2, Eq. (15)"},{"comment":"The closest scene-graph-augmented baselines are missing from the experiments. CCoT [35], KM-COT [36], BDoG [54], and LLaVA-SG [43] are described in Related Work as the most related approaches, but none of them appears in Tables 1 or 2 or in the main-results discussion. Without these comparisons, the claim in Section 4.4 that ISGR 'reaches state-of-the-art results' on multimodal question answering is not supported; the observed gains over LLaVA-v1.5 could in principle be due to the larger SFT data mixture or the query-generation mechanism rather than the proposed interaction graph construction. Please add these baselines with matched training data and evaluation settings, or explicitly temper the SOTA claim.","section":"Sections 2.3 and 4.3–4.4"},{"comment":"The training signal for the interaction graphs has a potential circularity that is not addressed. The interaction-augmented graphs used as SFT targets and as part of the reward ground truth are generated by an unspecified multimodal LLM through the ICoT prompts in Section 3.2. Section 3.3.1 mentions 'manually verified interaction triplets' but does not specify which model produced the graphs, how many annotators were involved, what the agreement was, or how much of the 300K interaction data was manually verified. If the generator is from the same model family as the model being trained (or shares the same weights), the learned 'long-term memory' may simply reproduce generator biases rather than learn generalizable interaction reasoning. Please specify the generator model(s), the verification protocol, and any quality-control statistics.","section":"Sections 3.2 and 3.3.1"},{"comment":"All reported benchmark numbers appear to come from a single run or a single evaluation, with no error bars, multiple seeds, or statistical significance tests. This is especially problematic for Table 3, where the average performance differences between the full reward and each single-component ablation are at most 0.7 points (59.98 vs. 59.48 vs. 59.40 vs. 59.26) — well within typical run-to-run variance for 7B VLMs. The headline IRR gains in Table 1 and Figure 3 should be accompanied by variance estimates or, at minimum, results from multiple seeds, to establish that the IRR contribution is not noise.","section":"Tables 1–3 and Section 4.4"}],"minor_comments":[{"comment":"In the abstract, 'lone-term memory reinforcement' should be 'long-term memory reinforcement.'","section":"Abstract"},{"comment":"The text says 'derived from our ICoT aproach'; 'aproach' should be 'approach.'","section":"Section 3.3.1"},{"comment":"The set notation in Eq. (12) has mismatched braces: D_interact = {(I_i, Q_i, Gfinal_i)}^N should be {(I_i, Q_i, Gfinal_i)}_{i=1}^N.","section":"Eq. (12)"},{"comment":"The dataset size numbers are internally inconsistent: the small-scale variant is stated as 841K, but LLaVA-v1.5-mixed-665k (665K) plus 300K interaction-augmented data sums to 965K; the medium-scale variant is stated as 1,371K, but the listed components sum to 1,441K. Please clarify deduplication and the exact composition of each variant.","section":"Table 4 and Section 4.1"},{"comment":"In Algorithm 1, the loop variable 'I' conflicts with the use of 'I' for the image; using a lowercase index (e.g., 'i') would avoid the confusion.","section":"Algorithm 1"},{"comment":"Reference [54] is incomplete, containing '[n. d.]' and missing the full proceedings information; also, the text uses 'KM-COT' while the reference list entry [36] is titled 'Kam-cot,' so the naming should be unified.","section":"References"},{"comment":"The figure contains informal elements such as 'IRR✨' and uses 'ICOT' in one label instead of the consistent 'ICoT'; please clean up the figure labels.","section":"Figure 2"}],"recommendation":"major_revision","confidential_remarks":"The manuscript appears to target ACM Multimedia 2025 (as indicated by the footer), yet two of the closest related methods, CCoT and BDoG, were published in the same venue/community but are not included as baselines; a reviewer or editor may want to consider whether the citation and baseline choices fully acknowledge prior work. The undefined reward function in Eq. (15) is the most serious issue: without it, the IRR component of the central claim is unverifiable, and the section must be rewritten with full specifications before the paper can be considered for acceptance. The dataset-size inconsistencies in Table 4 also suggest that the experimental details need careful re-checking before resubmission."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nHere's the short version: ISGR is a sensible pipeline for adding interactional reasoning to VLMs, and the reported gains are consistent enough that the paper deserves a real look. But the central new mechanism, the IRR reward, is never actually specified, and the evidence that the RL stage helps is thinner than the numbers suggest.\n\nWhat's new: the specific combination of summarize-and-align graph construction with an interactional chain-of-thought, followed by GRPO-style reinforcement using an interaction-focused reward. I don't see this exact combo in the cited literature. The paper also does something right in evaluation: the gains on SEEDBench, MMT-Bench, and RealWorldQA are non-trivial, and the ablations are reasonably systematic. The case studies are illustrative, and the overall framing is coherent.\n\nNow the soft spots, in order of severity.\n\nFirst, Eq. 15. The reward is written as a weighted sum of three functions—F_focus, F_disamb, F_rele—and those functions are never defined. The surrounding text says the reward evaluates graph quality and answer correctness, but none of the three terms takes the ground-truth answer as input. Since IRR is a headline contribution, this is a load-bearing gap. A reader cannot reproduce or even precisely understand what the RL stage is optimizing.\n\nSecond, Table 3: removing any one reward component changes the average by less than one point. Without error bars or multiple seeds, that's within run-to-run noise. So the claim that the reward components matter is not supported by the reported numbers.\n\nThird, missing baselines. The paper compares against a set of general VLMs, but not the closest prior work in scene-graph-augmented reasoning: CCoT, KM-COT, BDoG, LLaVA-SG are all cited in the related work but absent from the experiments. So the state-of-the-art claim is not backed up.\n\nFourth, the data generation loop. The interaction graphs used for SFT and reward are produced by an ICoT process using a multimodal LLM, and the paper doesn't say which model or how much manual verification was done beyond a vague \"manually verified interaction triplets.\" If the graphs come from the same model family being trained, there's a real circularity risk.\n\nThe code is not released yet, which makes the missing definitions harder to check.\n\nWho this is for: someone working on scene-graph-based multimodal reasoning or RL for VLMs. It's not a field-reorganizing result, but it's a plausible direction.\n\nMy recommendation: send it to peer review. The core idea is worth referee time, and the issues—reward definition, error bars, baselines, data provenance—are addressable. I wouldn't cite it in its current form.","headline":"A coherent interaction-scene-graph pipeline with consistent benchmark gains, but the reward function at the core of the claimed contribution is never defined and the evidence for the RL stage is thin.","tokens_in":18311,"tokens_out":2541,"would_cite":false,"duration_ms":23120,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Training vision-language models on action-aware scene graphs—who does what to whom—improves interaction-heavy visual question answering by up to 7.9 percentage points over a standard LLaVA-v1.5 baseline.","keywords":["scene understanding","interactional reasoning","vision-language models","scene graph generation","visual question answering","reinforcement learning","chain-of-thought","interaction-augmented scene graph"],"falsifier":"Regenerate the 300K interaction-augmented scene graphs with a different vision-language model generator, or shuffle the subject–action–object triples while keeping all other training data and rewards fixed; if the SEEDBench and MMT-Bench gains survive the shuffle, the graphs themselves are not the causal mechanism. A more direct version: build a small human-annotated interaction-graph test set and check whether models trained on ICoT graphs agree with humans more than a LLaVA-v1.5 baseline does.","tokens_in":17313,"feed_emoji":"🕸️","tokens_out":10729,"duration_ms":96652,"temperature":0.7,"pith_summary":"The paper tries to establish that vision-language models reason better about dynamic visual interactions—who is catching, throwing, blocking, or kicking—when they are trained on scene graphs that contain functional relationships in addition to spatial ones. It proposes ISGR, a framework that builds these interaction-augmented graphs through a summarize-and-align process and an interactional chain-of-thought, converts the graphs into training questions, and then reinforces the model with a reward signal for focus, disambiguation, and relevance. If the claim is right, a standard 7B model can gain roughly 7–8 percentage points on interaction-heavy benchmarks, match or surpass a much larger training set, and reach state-of-the-art multimodal question answering without changing its architecture.","feed_headline":"Action-aware scene graphs lift AI vision scores by 7.9 points","feed_subtitle":"Teaching models who does what to whom adds 7.8 and 7.1 points on MMT-Bench and RealWorldQA.","key_machinery":"The load-bearing object is the interaction-augmented scene graph: a directed graph whose edges are subject–action–object triples grounded in bounding boxes, built by the Interactional Chain-of-Thought (ICoT) over an abstracted spatial graph. ICoT proceeds through subject identification, action recognition, object identification, and relation formalization, then applies saliency, grounding, and consistency constraints. The graph does triple duty: it is the supervision target for SFT, the raw material for four query types that generate object-object, subject-relation, relation-object, and comprehensive questions, and the input that the reward function scores with $R(y_k)=\\lambda_1 F_{\\text{focus}}(y_k)+\\lambda_2 F_{\\text{disamb}}(y_k)-\\lambda_3 F_{\\text{rele}}(y_k)$, with weights 0.4, 0.4, and 0.2. This triple role is what unifies graph construction and vision-language inference in a single optimization loop.","core_discovery":"The central claim is that the bottleneck in scene reasoning is not spatial layout but functional interaction, and that this bottleneck can be removed through data and reward. The authors report that their full model, ISGR(SFT+IRR), outperforms the LLaVA-v1.5 baseline by +7.9% on SEEDBench, +7.8% on MMT-Bench, and +7.1% on RealWorldQA, and state that it reaches state-of-the-art results on multimodal question answering. The mechanism is a two-stage memory reinforcement: supervised fine-tuning on 1.3M samples that include 300K interaction-augmented scene graphs, followed by interaction reasoning reinforcement on just 500 curated interaction instructions using a reward composed of focus ($\\lambda_1=0.4$), disambiguation ($\\lambda_2=0.4$), and relevance ($\\lambda_3=0.2$) terms. Because the same graphs serve as SFT targets, as the source of four query types, and as the grounding for rewards, construction and reasoning are optimized in one loop.","pith_inferences":["Editorial: If the ICoT graphs come from a strong external vision-language model, then ISGR is effectively a distillation of that model's interaction knowledge into an open 7B model; a quick way to test this is to compare ISGR(SFT) against a model trained on the same generator's captions without graph structure.","Editorial: The +31.4% gain on Instance Counting suggests that interaction role labels help the model individuate otherwise similar objects (e.g., players distinguished by role); this predicts improvements on dedicated counting and multi-object tracking benchmarks that the paper does not evaluate.","Editorial: The reward decomposition—focus, disambiguation, relevance—targets failure modes that appear in other structured reasoning outputs, so the same reward recipe could be tested on temporal event chains in video or on relation extraction from documents, not just static images."],"forward_implications":["No architectural change is needed: a standard LLaVA-v1.5 7B model absorbs interaction reasoning through SFT plus a small reward-tuned stage, so the method can be applied to any instruction-tuned vision-language model.","Interaction-focused supervision is data-efficient: ISGR(SFT)-S, trained on 0.8M samples, stays competitive with the 1.2M LLaVA baseline, suggesting richer relational labels extract more learning per example.","Gains concentrate where interactions matter: on SEEDBench, Instance Interaction improves +7.3% and Instance Counting +31.4%, while Spatial Relation and Text Understanding decline slightly, showing the model trades some positional precision for functional understanding.","A small reward set goes far: 500 curated interaction instructions added through interaction reasoning reinforcement yield the largest jumps on RealWorldQA (+7.1), MMT-Bench (+7.8), and SEEDBench (+7.9) relative to the baseline.","Spatial grounding contributes consistently: removing bounding boxes from the graphs costs 0.5–2.2 points across benchmarks, though the reinforced model still beats the SFT-only model without boxes on most tasks."],"supporting_citations":[{"why":"Provides the LLaVA-v1.5 baseline architecture and the LLaVA-v1.5-mixed-665k dataset that forms the base of both SFT variants.","marker":"[30]"},{"why":"Supplies 176K OpenImages samples with manually annotated scene graphs used for the interaction-augmented dataset.","marker":"[20]"},{"why":"Supplies the LVIS-Instruct-4V instruction set included in the 1.3M SFT variant.","marker":"[44]"},{"why":"Supplies the LRV-Instruct instruction set included in the 1.3M SFT variant.","marker":"[29]"},{"why":"SEEDBench is the benchmark on which the largest reported gain (+7.9%) is measured.","marker":"[22]"},{"why":"RealWorldQA is the benchmark on which the model reports +7.1% over the baseline.","marker":"[47]"},{"why":"MMT-Bench is the benchmark on which the model reports +7.8% over the baseline.","marker":"[49]"},{"why":"GQA is used to show that spatial reasoning also improves (+1.4% for the SFT model) and that reinforcement trades off some of that gain.","marker":"[16]"}],"fun_headline_variants":["Interaction-aware graphs add 7.9 points to AI vision","Why VLMs need object interactions: 7.9-point gain","Boost AI vision with interaction memory: +7.9 points","Teaching AI object interactions lifts vision scores by 7.9"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole training signal rests on the assumption that the interaction graphs produced automatically by the ICoT process are accurate enough to teach real interaction reasoning, since the paper only says these graphs were 'manually verified' without saying how much or by whom.","fun_headline_variants_meta":{"raw":{"variants":["Interaction-aware graphs add 7.9 points to AI vision","Why VLMs need object interactions: 7.9-point gain","Boost AI vision with interaction memory: +7.9 points","Teaching AI object interactions lifts vision scores by 7.9"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001406,"raw_usage":{"total_tokens":5701,"prompt_tokens":980,"completion_tokens":4721,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":596,"completion_tokens_details":{"reasoning_tokens":4649}},"tokens_in":596,"tokens_out":4721,"duration_ms":35294,"temperature":1.0,"reasoning_tokens":4649,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T21:38:57.293145+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Regenerate the 300K interaction-augmented scene graphs with a different vision-language model generator, or shuffle the subject–action–object triples while keeping all other training data and rewards fixed; if the SEEDBench and MMT-Bench gains survive the shuffle, the graphs themselves are not the causal mechanism. A more direct version: build a small human-annotated interaction-graph test set and check whether models trained on ICoT graphs agree with humans more than a LLaVA-v1.5 baseline does.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"RealWorldQA is the benchmark on which the model reports +7.1% over the baseline."}],"review_version":1}