Pith. sign in

REVIEW 2 major objections 5 minor

Scene graphs let multimodal models navigate visual scenes instead of hunting isolated objects.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-11 03:11 UTC pith:LZNX5N5M

load-bearing objection Solid systems paper: hierarchical scene-graph data + node-as-proxy GRPO delivers large, ablated gains on fine-grained and spatial MLLM benchmarks; residual risk is teacher-graph bias, already partly audited. the 2 major comments →

arxiv 2607.05716 v3 pith:LZNX5N5M submitted 2026-07-07 cs.CV

Scene Graph Thinking: Reinforcing Structured Visual Reasoning for Multimodal Large Language Models

Zhiwei Yang , Yuanchen Wu , Nan Zhang , Yucong Meng , Ke Yan , Shouhong Ding This is my paper
classification cs.CV
keywords scene graph thinkingmultimodal large language modelsstructured visual reasoningnode-as-proxy rewardsGRPOfine-grained perceptionhierarchical scene graphschain-of-thought
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Current multimodal large language models still treat images as bags of isolated objects. When they need a small detail or a spatial relation they crop and zoom, but they lack an explicit map of how entities connect. This paper claims that building hierarchical scene graphs—nodes for objects and sub-parts with attributes, boxes and depth, edges for spatial, interaction and semantic links—then sampling reasoning traces from those graphs and training with node-as-proxy rewards turns the model into a structured navigator. The result is large gains on fine-grained perception, high-resolution, and 2-D/3-D spatial benchmarks while still generalizing to ordinary vision tasks. A sympathetic reader cares because the same models that already see well can now follow relational pathways instead of heuristic search, which is exactly what complex visual questions require.

Core claim

An automated pipeline that turns flat image–text pairs into hierarchical, depth-aware scene graphs, followed by supervised fine-tuning on 120K graph-sampled chain-of-thought traces and GRPO reinforcement with node-grounded and node-relevance rewards, internalizes structured visual reasoning so that even 3B- and 7B-parameter models substantially outperform their baselines and several larger or tool-using systems on fine-grained and spatial benchmarks.

What carries the argument

Node-as-proxy graph rewards: instead of supervising ambiguous edges, the method rewards trajectories that visit visually grounded, query-relevant nodes (node-grounded reward checks crop–description match; node-relevance reward scores semantic support of anchored nodes), thereby consolidating efficient graph exploration.

Load-bearing premise

The hybrid automatic graphs (large teacher MLLM plus depth and segmentation models, then two-round filtering) are accurate and complete enough that the sampled traces and LLM-as-judge rewards teach real structured navigation rather than artifacts of the teachers.

What would settle it

Replace the graph-derived cold-start data and node rewards with equal-volume ordinary VQA or random-crop traces; if the VStarBench and CVBench gains largely disappear, the structured-graph claim fails.

Watch this falsifier — get emailed when new claim-graph text bears on it.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The paper proposes Scene Graph Thinking (SaGe), a paradigm that equips MLLMs with structured visual reasoning by converting flat image–text pairs into hierarchical scene graphs (compositional nodes with attributes/bboxes/depth, multi-type edges), sampling 120K node- and edge-centric CoT traces, and applying two-stage post-training: SFT cold-start followed by GRPO with complementary node-as-proxy rewards (node-grounded visual alignment and node-relevance to query intent). Built on Qwen2.5-VL-3B/7B, SaGe reports large gains on high-resolution fine-grained benchmarks (VStarBench to 89.0, +12–13 over baselines; HRBench-4K/8K), spatial tasks (CVBench-2D/3D), and modest gains on MMStar, RefCOCO, and ChartQA, with ablations isolating SFT, CoT tags, and each reward component, plus a teacher-surpassing comparison.

Significance. If the gains hold under independent re-implementation, the work supplies a practical, scalable recipe for injecting relational structure into MLLM reasoning without multi-turn tool loops. Strengths include a fully automated hybrid graph engine, publicly released code, systematic ablations (Tables 3–5) that isolate SFT, GRPO, CoT tags and NGR/NRR, monotonic data-scale/composition curves (Fig. 6), and the result that SaGe-7B exceeds its Qwen2.5-VL-72B teacher on several metrics (Table 6). The node-as-proxy design is a clean alternative to brittle edge-level supervision and is of clear interest to the fine-grained multimodal community.

major comments (2)
  1. §3.2–3.3 and Appendix B: node/edge precision under Gemini-3-Pro is high (94.5%/92.9%), yet open-vocabulary recall and hierarchical completeness remain unquantified. Because the central claim is that the model learns genuine structured navigation rather than teacher artifacts, a short recall or coverage analysis (e.g., fraction of human-annotated entities recovered on a held-out subset) would strengthen the weakest assumption without altering the experimental design.
  2. §3.4 Eqs. (5)–(7) and Appendix D: both NGR and NRR rely on external LLM/MLLM judges (Qwen3 family). While Table 5 shows complementarity and Table 6 shows student > teacher, a brief sensitivity check (different judge model or human agreement on a reward subsample) would confirm that the reported gains are not judge-specific.
minor comments (5)
  1. Figure 1 caption and §1: the contrast with “isolated crops + heuristic search” is clear, but a short quantitative comparison of navigation length or number of hops versus DeepEyes/Pixel-Reasoner would make the efficiency claim more concrete.
  2. Table 1: FSP/FCP columns for proprietary models are often blank; a note explaining missing entries would avoid the appearance of selective reporting.
  3. §4.1 Implementation: the 26K RL mix (2K grounding + 4K chart + 20K V*) is reasonable, yet the exact sampling ratios and whether any V* test leakage is possible should be stated explicitly.
  4. Appendix C multi-criteria caption filter (Fig. 14): the eight binary rules are useful; reporting the fraction of captions discarded at each stage would help readers gauge data-construction cost.
  5. Notation: vt is defined with ct, at, bt, dt in Eq. (3), yet later text sometimes uses “entity” interchangeably; a single glossary of tags (<entity>, <bbox>, <depth>) would improve readability.

Circularity Check

0 steps flagged

No significant circularity: empirical ML method with external data generation and independent public-benchmark evaluation.

full rationale

This is a standard empirical multimodal training paper. Scene graphs are constructed via an automated hybrid pipeline (Qwen2.5-VL-72B + Depth-Anything + SAM), 120K CoT samples are LLM-sampled from those graphs, SFT + GRPO with node-as-proxy (LLM-as-judge) rewards are applied, and gains are reported on eight held-out public benchmarks (VStarBench, HRBench-4K/8K, CVBench-2D/3D, MMStar, RefCOCO, ChartQA). None of the load-bearing performance claims reduce by construction to fitted constants, self-defined quantities, or self-citation uniqueness theorems. Teacher-model artifacts are a validity concern (addressed by the authors via Gemini-3-Pro audit, two-round filtering, and Table 6 teacher-surpassing results) but do not constitute circularity under the defined patterns. Self-citations (e.g., prior Yang et al. CLIP/Depth works) are peripheral and non-load-bearing. The derivation chain is self-contained against external metrics; score 0 is the correct honest finding.

Axiom & Free-Parameter Ledger

4 free parameters · 3 axioms · 2 invented entities

Standard empirical ML paper. Free parameters are ordinary training hyper-parameters and discrete reward tiers. Axioms are the usual domain assumptions that large VLMs can serve as reliable annotators and judges. The main invented constructs are the SaGe paradigm itself and the two node-proxy rewards; both are operationally defined and evaluated, not postulated as new physical entities.

free parameters (4)
  • SFT learning rate / batch size
    1e-5, batch 256; chosen by authors, not derived.
  • RL learning rate / batch / rollouts
    1e-6, batch 128, 8 rollouts; standard GRPO knobs.
  • Node-relevance reward tiers (1.0 / 0.8 / 0.5 / 0.2 / 0.0)
    Hand-designed discrete levels for semantic match quality; affect the policy gradient.
  • KL coefficient β and clip ε in GRPO
    Standard PPO-style hyper-parameters left at conventional values.
axioms (3)
  • domain assumption Qwen2.5-VL-72B + Depth-Anything + SAM produce sufficiently accurate hierarchical nodes and multi-type edges for training data.
    Core of the automated data engine (Sec. 3.2); validated only by post-hoc Gemini precision numbers.
  • domain assumption LLM/MLLM judges can reliably score node-grounded visual alignment and node-relevance to query intent.
    Used for both reward signals in GRPO (Sec. 3.4 and App. D).
  • ad hoc to paper Node sequences are an adequate proxy for valid graph trajectories; edge-level supervision is unnecessary.
    Explicit design choice stated in Sec. 3.4 to avoid ambiguous multi-relation edges.
invented entities (2)
  • Scene Graph Thinking (SaGe) paradigm no independent evidence
    purpose: Overall framework that internalizes hierarchical scene graphs into MLLM reasoning.
    Name and packaging of the method; evaluated empirically.
  • Node-as-proxy graph rewards (NGR + NRR) no independent evidence
    purpose: Provide dense, verifiable supervision for GRPO without requiring edge labels.
    Novel reward design; ablated in Table 5.

pith-pipeline@v1.1.0-grok45 · 31830 in / 2643 out tokens · 28146 ms · 2026-07-11T03:11:00.726585+00:00 · methodology

0 comments
read the original abstract

Multimodal Large Language Models (MLLMs) have demonstrated strong perception and reasoning capabilities. However, most existing models focus on isolated objects and neglect structured relationships for efficient target navigation, limiting their performance on visually intensive tasks. To address this challenge, we introduce Scene Graph Thinking (SaGe), a novel paradigm that enables fine-grained and structured visual reasoning through explicit scene-graph representations. Specifically, we first introduce an automated data engine that converts flat image-text corpora into structured scene graphs, where hierarchical entities constitute the nodes and diverse visual relations define the edges. Building upon this, we construct 120K high-quality training data by sampling reasoning traces from scene graphs. Then, two-stage graph-aligned post-training paradigms are introduced, where supervised fine-tuning internalizes MLLMs with structured reasoning, and subsequent reinforcement fine-tuning proposes node-as-proxy graph rewards to consolidate efficient graph exploration. With curated data and graph-aligned training, our approach achieves significant improvements across eight multimodal benchmarks, demonstrating strong effectiveness on fine-grained perception and reasoning tasks. Code is available at https://github.com/zwyang6/SaGe.

Figures

Figures reproduced from arXiv: 2607.05716 by Ke Yan, Nan Zhang, Shouhong Ding, Yuanchen Wu, Yucong Meng, Zhiwei Yang.

Figure 1
Figure 1. Figure 1: Our motivation. (a) Previous methods overlook struc￾tured relationships within the scene, leading to inefficient target navigation and suboptimal performance. (b) The proposed SaGe internalizes structured scene graphs to enable fine-grained visual reasoning, achieving more efficient and reliable performance. tence in multimodal perception (Lai et al., 2024; Liu et al., 2025) understanding (Wang et al., 202… view at source ↗
Figure 2
Figure 2. Figure 2: Scene Graph Construction. We convert each flat image–text pair into a hierarchical scene graph. Given an image Ii, (a) we first perform Compositional Node Mining, modeling all objects and their sub-components as nodes, each equipped with bounding boxes and attribute annotations. (b) We then build Depth-aware Node Modeling by assigning depth cues to each node. (c) Finally, edges are constructed based on nod… view at source ↗
Figure 3
Figure 3. Figure 3: Training Data Sampling and Verification. We adopt (a) node- and edge-centric query sampling with a (b) two-round data verification strategy, (c) yielding 120K CoT-augmented training samples. we use it to prompt SAM for precise segmentation. After boundary erosion, we extract depth values within the mask to estimate a depth range for each entity, providing each node with reliable depth cues for 3D-aware rea… view at source ↗
Figure 4
Figure 4. Figure 4: Two-stage Training Pipeline. We first adopt the (a) supervised fine-tuning stage to internalize the MLLMs with graph-aligned reasoning. Then (b) a reinforcement learning stage is conducted with proxy-as-node graph rewards, consolidating the structured reasoning. GRPO-based reinforcement learning. We contend that defin￾ing a verifiable edge-level supervision signal is inherently ambiguous, as a single pair … view at source ↗
Figure 5
Figure 5. Figure 5: Case study on various QA scenarios. Compared to baseline, our method generates correct answers with node-articulated CoT [PITH_FULL_IMAGE:figures/full_fig_p007_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Data Scaling and Composition Analysis. Base: ground￾ing and counting. Cap.: caption. Mult.: multi-hop reasoning data. level supervision without trace-level guidance may lead to reward hacking, where the model attends to visually salient but target-irrelevant nodes. When combined, NRR and NGR exhibit strong complementarity: NRR guides which nodes to traverse, while NGR ensures the correctness of the visited… view at source ↗
Figure 7
Figure 7. Figure 7: Node Error Breakdown. (a) Distribution of node errors. (b) Number distribution of different error types. The nodes constructed in our scene graph achieve an accuracy of 94.5%, demonstrating the high quality of our scene graph. 55.6% （1273） 14.0% （320） 15.2% （347） 15.3% （350） Wrong direction Wrong desciption Wrong relation type Invalid relation 1273 350 347 320 0 200 400 600 800 1000 1200 1400 Edge Error Br… view at source ↗
Figure 8
Figure 8. Figure 8: Edge Error Breakdown. (a) Distribution of edeg errors. (b) Number distribution of different error types. The edges in our constructed scene graph attain an accuracy of 92.9%, which validates the reliability and quality of the scene graph construction. 13 [PITH_FULL_IMAGE:figures/full_fig_p013_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Case study on various QA scenarios. Our method demonstrates strong spatial understanding and fine-grained perception through node-articulated CoT, highlighting the effectiveness of our scene-graph–based structured reasoning. B.2. More Visualized QA Cases As illustrated in [PITH_FULL_IMAGE:figures/full_fig_p014_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Illustration of attribute-centric samples. We generate 25K attribute-centric samples, comprising 10K color-related, 5K state-related, 5K shape-related, and 5K material-related instances. Each sample is accompanied by a node-articulated CoT, providing explicit evidence to support structured reasoning. 16 [PITH_FULL_IMAGE:figures/full_fig_p016_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Illustration of spatial reasoning, grounding, and counting samples. We include 30K spatial reasoning samples, covering both 2D and 3D spatial reasoning tasks. The traceable evidence within the <bbox> and <depth> tags provides spatial clues to support reasoning. To further enhance data diversity, we additionally generate 10K grounding samples and 10K counting samples. 17 [PITH_FULL_IMAGE:figures/full_fig_… view at source ↗
Figure 12
Figure 12. Figure 12: Illustration of multi-hop navigation and local caption samples. We generate 25K samples with multi-hop reasoning traces, together with 10K local captions. The local caption data focus on detailed descriptions of specific regions, such as a bounding box or half of the image. 18 [PITH_FULL_IMAGE:figures/full_fig_p018_12.png] view at source ↗
Figure 13
Figure 13. Figure 13 [PITH_FULL_IMAGE:figures/full_fig_p019_13.png] view at source ↗
Figure 14
Figure 14. Figure 14: Multi-criteria scoring scheme for captioning. We employ a multi-criteria scoring scheme that emphasizes node coverage and edge accuracy, retaining only high-consistency caption samples. 20 [PITH_FULL_IMAGE:figures/full_fig_p020_14.png] view at source ↗
Figure 15
Figure 15. Figure 15: Judge prompt of the node-grounded reward. More detailed examples used to instruct the MLLM are omitted for brevity. 21 [PITH_FULL_IMAGE:figures/full_fig_p021_15.png] view at source ↗
Figure 16
Figure 16. Figure 16: Judge prompt of the node-relevance reward. More detailed examples used to instruct the LLM are omitted for brevity. 22 [PITH_FULL_IMAGE:figures/full_fig_p022_16.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.