REVIEW 4 major objections 4 minor 1 cited by
Schema-Guided Scene-Graph Reasoning based on Multi-Agent Large Language Model System
T0 review · 4 major / 4 minor · reviewed 2026-08-09 · deepseek-v4-flash
Pith's one-line read SG2, a schema-guided multi-agent LLM framework, outperforms full-graph prompting and ReAct-style retrieval on scene-graph numerical Q&A and planning tasks.
desk verdict A solid engineering contribution with a real ablation, but the 'beats ReAct everywhere' claim overreaches the VirtualHome evidence. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the scene graph schema $S$, a textual description of node types, attributes, and edge semantics that is provided to both modules while the graph instances are withheld. The schema does two jobs: it lets the Task Planner reason abstractly and emit queries whose wording matches the schema, and it lets the Code Writer compose low-level graph APIs into arbitrary Python programs that filter the graph to the query's needs. A second mechanism is the multi-agent context split: the Task Planner receives only retrieved subgraph summaries, the Code Writer sees only the current query and execution errors, and a Verifier decides whether the query has been addressed. Together these keep each prompt free of information the agent's role does not require.
What would settle it
Construct a new scene graph and task in which the schema deliberately omits one attribute or relationship that is present in the graph and required to solve the task; if SG2 then fails while a full-graph-prompting baseline succeeds, the schema-completeness premise is the limiting factor. A complementary check is to hold the graph fixed and vary only the schema: if success changes when the schema text changes, the mechanism is genuinely schema-driven rather than graph-data-driven.
Extended reading notes
Core claim
The paper's central claim is that the scene graph schema is both sufficient and beneficial for grounded spatial reasoning: the Reasoner can plan abstractly from the schema alone, generate natural-language queries aligned with it, and never needs the raw graph, while the Retriever translates each query into a program that extracts exactly the relevant subgraph. This turns graph access into a program-synthesis problem and keeps irrelevant nodes out of the context window. The reported results back this up: 98% success on BabyAI numerical Q&A versus 86% for ReAct, 96-97% on traversal planning versus 94-95%, and perfect scores on VirtualHome household planning where all baselines fail at least one of eight tasks. The ablation against ReAct-limit and SG2-limit attributes the gain to two design choices acting together: programmatic query-driven retrieval and the separation of reasoning and retrieval contexts.
Load-bearing premise
The load-bearing premise is that the scene graph schema is complete and precise enough that the Reasoner can ask for every piece of information it will need without ever seeing the graph, and the Code Writer can turn each query into a correct program from the schema alone.
Editorial extensions
If this is right
- LLM reasoning over large scene graphs no longer requires serializing the whole graph into the prompt, so context length stops scaling with environment size.
- A fixed, limited set of graph APIs becomes competitive when paired with the multi-agent Reasoner-Retriever split, reducing sensitivity to tool quality.
- Smaller language models benefit more from schema-based abstract reasoning than from reading full textualized graphs; with Phi4-14B, SG2 doubles the best baseline's success rate on the numerical Q&A task.
- The framework generalizes across two different environments, a 2D grid world and a large indoor simulator, suggesting it is not tied to one graph format.
Reading between the lines
- The ceiling of the approach is schema completeness: any attribute or relationship omitted from $S$ is invisible to the Reasoner, so a task solvable by a full-graph model could become unsolvable by SG2 even though the answer is in the graph.
- The same schema-plus-code pattern could transfer to other structured data sources with documented schemas, such as relational databases or knowledge graphs, where the 'graph' is not spatial.
- A testable extension is scaling graph size while holding task difficulty fixed; the token-count analysis suggests SG2's advantage over full-graph prompting should grow with graph size, but the paper does not run that controlled comparison.
- The Verifier checks whether code output addresses the query, not whether the extracted values are semantically correct, so a confidently wrong program could pass verification; a stronger verifier that cross-checks answers against the raw graph would close this gap.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes SG2, a schema-guided multi-agent LLM framework for reasoning over scene graphs. The Reasoner module (Task Planner, Tool Caller) decomposes the task, issues natural-language queries, and calls reasoning tools, while the Retriever module (Code Writer, Verifier) translates those queries into executable Python programs over the scene graph. Both modules are conditioned on the scene graph schema rather than on the full graph data, and they alternate iteratively. Experiments are reported in BabyAI (numerical Q&A and traversal planning) and VirtualHome (household planning), comparing against graph-prompting baselines, SayPlan, and ReAct. Additional ablations isolate the contributions of API capacity versus multi-agent context separation, and smaller models are evaluated on the BabyAI Q&A task. The reported tables show SG2 achieving the highest success rate on all tasks, but several statistical and comparison gaps weaken the paper's headline claim.
Significance. If the results hold, SG2 is a useful contribution: it avoids serializing large scene graphs into LLM prompts, separates reasoning and retrieval contexts to mitigate context pollution, and gives concrete evidence for the value of the multi-agent split through the ReAct-limit versus SG2-limit ablation. The prompt templates, qualitative traces, and compute analysis are welcome transparency. The main caveats are that the VirtualHome comparison is underpowered and omits ReAct/SayPlan, and that the BabyAI traversal margin over ReAct is a single point in one condition. The central architectural claim is defensible, but the paper's 'all evaluations' phrasing needs to be scoped and supported with statistical care.
major comments (4)
- [Household Task Planning, Table 2] The VirtualHome evidence for SG2 is statistically fragile. Only 8 tasks are tested per environment; SG2 achieves 8/8 while the best baselines achieve 7/8 or 6/8. Under a simple binomial model, a 7/8 versus 8/8 difference is not compelling, and a single task flip would produce a tie. The paper reports no confidence intervals, no repeated task samples, and no significance tests. This is a load-bearing issue because the VirtualHome result is used to support the claim that SG2 'outperforms ... in all evaluations.' Please add exact binomial confidence intervals or otherwise demonstrate that the margin is not sampling noise.
- [Household Task Planning, Experimental Settings] ReAct and SayPlan are not evaluated in VirtualHome, so the Introduction's claim that SG2 outperforms 'ReAct and graph prompting baselines in all evaluations' is not established for that environment. The stated reason that ReAct and SayPlan 'do not work well without the demonstrations' may justify the omission, but it is an unverified assertion and should not be presented as a substitute for an experiment. At minimum, the claims should be scoped to the environments and baselines actually evaluated, and the omission should be acknowledged explicitly.
- [Table 1, Trv-2 row] In the BabyAI traversal task, the SG2-versus-ReAct difference is 96% versus 95% in Trv-2 and 97% versus 94% in Trv-1; with 100 instances per task and no repeated runs, these differences are within sampling noise. The later ablation in Table 3 provides stronger support for the multi-agent design, since SG2-limit outperforms ReAct-limit by large margins (e.g., 70% versus 11% in Trv-2). The paper should therefore present the design benefit primarily through that controlled comparison and avoid relying on the one-point Trv-2 gap when claiming a large improvement over ReAct.
- [Method, Retriever Module] The framework depends on the completeness of the scene graph schema S. Because the Task Planner is explicitly never shown the graph data and must formulate all queries from S, any attribute or relationship missing from S is unrecoverable, no matter how many queries are issued. The paper does not test or analyze robustness to incomplete, noisy, or under-specified schemas, which is a real correctness risk for the claimed generality of the approach. Please add a discussion of this limitation or a small experiment that ablates schema fields to show how performance degrades.
minor comments (4)
- [Retriever Module] The prose describing the Verifier appears inverted relative to the prompt template in Table 5. The text says 'If the query is deemed addressed, it prompts the Code Writer to re-write the code. Otherwise, it summarizes the result and sends it back to the Reasoner,' but Table 5 says addressed or partially answered queries should be summarized and only NOT ADDRESSED responses should trigger rewriting. Please make the description consistent.
- [Experimental Settings] Please clarify how planning success is determined in VirtualHome. The paper states that all experiments are static, with methods generating solutions 'solely based on the initial scene graph without interacting with the environment,' so it is unclear whether the 'desired outcome' for household tasks is checked by executing the plan in the VirtualHome simulator, by evaluating the final state predicate, or by manual inspection.
- [SLM experiments, Figure 5] The small-model evaluation uses only 20 trials per method and reports no confidence intervals. Given the large apparent differences (e.g., SG2 at 60% versus the best baseline at 30% for Phi4-14B), please report the raw counts or binomial confidence intervals so the reader can assess the reliability of these comparisons.
- [Throughout] Typos and formatting issues should be corrected in a revision, including 'epoches' (should be 'environments' or 'graph instances'), 'curley brackets', 'retrive', 'irrelevent', 'messagea' in Table 6, and 'Resutls' in section headings.
Circularity Check
No significant circularity: the SG2 claim is an empirical benchmark comparison against external baselines, and the VirtualHome baseline gap is an evidence-scope issue rather than a self-referential derivation.
full rationale
The paper contains no derivation chain in which a predicted quantity is defined from, or fitted to, the quantity it is said to predict. The method takes a task instruction I, a scene graph G, and a schema S, and produces a solution A = f(I, G, S; LLMs). The output is scored against ground-truth answers and action sequences provided by the BabyAI and VirtualHome environments, and against external baselines such as ReAct, SayPlan, CoT, BAG, and Algorithmic Prompting. This makes the evaluation self-contained and externally falsifiable. The schema S is an input to the Reasoner, but it does not determine the answer: the Reasoner must issue queries, the Retriever must execute code against actual graph data, and the final plan is verified against environment-defined success criteria. The few-shot demonstrations are held out from the tested tasks (e.g., Trv-1 examples are used for training and Trv-2 is the extrapolation test; two VirtualHome tasks are used as examples while the other eight are tested), so the reported successes are not demonstrations replayed by construction. Ablations such as SG2-limit versus ReAct-limit control for the API surface rather than defining the outcome in terms of the method's own outputs. The self-citations in the related-work sections (Chen et al. 2021; Chen et al. 2023) are not load-bearing for the central empirical claim. One stated limitation is that ReAct and SayPlan were not tested in VirtualHome: 'we do not test ReAct and SayPlan as they do not work well without the demonstrations.' This undercuts the breadth of the 'all evaluations' headline claim, and the VirtualHome comparison is based on only eight tasks per environment, but an omitted baseline and small sample size are statistical and completeness concerns, not circularity. No uniqueness theorem is imported from the authors, no ansatz is smuggled in via self-citation, and no known result is renamed as a new derivation. Overall, the paper's central claim rests on independent experiments rather than on a self-referential argument, so the circularity score is 0.
Assumptions & free parameters
assumptions (3)
- domain assumption The hand-authored schema S fully and faithfully describes all node types, attributes, and edge semantics needed for each task.
- domain assumption Retrieval code generated by the Code Writer executes and returns correct data after self-debugging, and the Verifier's addressed/not-addressed judgment is reliable.
- domain assumption The success rate over 100 trials (and 20 in the SLM study) gives a stable enough estimate to rank methods, and GPT-4o outputs at temperature 0 are effectively deterministic.
Cite this review
Pith. "Pith review of Schema-Guided Scene-Graph Reasoning based on Multi-Agent Large Language Model System." pith.science (2026). https://pith.science/paper/GUC4AMLU
@misc{pith2026250203450,
author = {Pith},
title = {Pith review of: Schema-Guided Scene-Graph Reasoning based on Multi-Agent Large Language Model System},
year = {2026},
howpublished = {\url{https://pith.science/paper/GUC4AMLU}},
note = {Machine review of arXiv:2502.03450}
}
read the original abstract
Scene graphs have emerged as a structured and serializable environment representation for grounded spatial reasoning with Large Language Models (LLMs). In this work, we propose SG^2, an iterative Schema-Guided Scene-Graph reasoning framework based on multi-agent LLMs. The agents are grouped into two modules: a (1) Reasoner module for abstract task planning and graph information queries generation, and a (2) Retriever module for extracting corresponding graph information based on code-writing following the queries. Two modules collaborate iteratively, enabling sequential reasoning and adaptive attention to graph information. The scene graph schema, prompted to both modules, serves to not only streamline both reasoning and retrieval process, but also guide the cooperation between two modules. This eliminates the need to prompt LLMs with full graph data, reducing the chance of hallucination due to irrelevant information. Through experiments in multiple simulation environments, we show that our framework surpasses existing LLM-based approaches and baseline single-agent, tool-based Reason-while-Retrieve strategy in numerical Q\&A and planning tasks.
Figures
Figures from the paper (5 more)
Forward citations
Cited by 1 Pith paper
-
Graphs Meet AI Agents: Taxonomy, Progress, and Future Opportunities
A survey that groups graph-empowered AI agent research into planning, execution, memory, and multi-agent coordination, plus agents-for-graphs and applications.
Reference graph
Works this paper leans on
-
[1]
Query mode. Reason on what information do you need to solve the task, and query for the information from the retriever
-
[2]
Call a function from the given function set to address a substep
Tool calling mode. Call a function from the given function set to address a substep
-
[3]
type": String. The type of the element type. Choices: root, room, agent, key, door, box, ball •
Solution mode. Give the solution to the task. [TOOL ANNOTATIONs] Please always format all your responses as follows: [Explanation] Explain your reasoning process succinctly. [Mode] Only one of QUERY or SOLUTION or TOOL. Note that the SOLUTION mode will terminate the conversation. So only use it when you have the entire solution. [Content] If QUERY, then g...
work page 2000
-
[4]
Autonomous Robots, 47(8): 1345–1365
Text2motion: From natural language instructions to feasible plans. Autonomous Robots, 47(8): 1345–1365. Liu, B.; Jiang, Y .; Zhang, X.; Liu, Q.; Zhang, S.; Biswas, J.; and Stone, P. 2023. Llm+ p: Empowering large language models with optimal planning proficiency. arXiv preprint arXiv:2304.11477. Liu, Y .; Li, D.; Wang, K.; Xiong, Z.; Shi, F.; Wang, J.; Li...
arXiv 2023
-
[5]
arXiv preprint arXiv:2310.08864
Open x-embodiment: Robotic learning datasets and rt-x models. arXiv preprint arXiv:2310.08864. Paranjape, B.; Lundberg, S.; Singh, S.; Hajishirzi, H.; Zettle- moyer, L.; and Ribeiro, M. T. 2023. Art: Automatic multi- step reasoning and tool-use for large language models. arXiv preprint arXiv:2303.09014. Press, O.; Zhang, M.; Min, S.; Schmidt, L.; Smith, N...
arXiv 2023
-
[6]
In The Twelfth International Conference on Learning Representations
Large language models as generalizable policies for embodied tasks. In The Twelfth International Conference on Learning Representations. Team, O. M.; Ghosh, D.; Walke, H.; Pertsch, K.; Black, K.; Mees, O.; Dasari, S.; Hejna, J.; Kreiman, T.; Xu, C.; et al
-
[7]
arXiv preprint arXiv:2405.12213
Octo: An open-source generalist robot policy. arXiv preprint arXiv:2405.12213. Wang, H.; Feng, S.; He, T.; Tan, Z.; Han, X.; and Tsvetkov, Y . 2023. Can Language Models Solve Graph Problems in Natural Language? In Thirty-seventh Conference on Neural Information Processing Systems. Wei, J.; Wang, X.; Schuurmans, D.; Bosma, M.; Xia, F.; Chi, E.; Le, Q. V .;...
arXiv 2023
-
[8]
Wu, S.-C.; Wald, J.; Tateno, K.; Navab, N.; and Tombari, F
How Easily do Irrelevant Inputs Skew the Responses of Large Language Models? In First Conference on Language Modeling. Wu, S.-C.; Wald, J.; Tateno, K.; Navab, N.; and Tombari, F
Show all 14 references
-
[11]
arXiv preprint arXiv:2308.07134, 4(5): 7
Natural language is all a graph needs. arXiv preprint arXiv:2308.07134, 4(5): 7. Yoran, O.; Wolfson, T.; Ram, O.; and Berant, J. 2023. Making retrieval-augmented language models robust to irrelevant context. arXiv preprint arXiv:2310.01558. Yu, W.; Gileadi, N.; Fu, C.; Kirmani...
2023 arXiv
-
[38]
Yao, S.; Zhao, J.; Yu, D.; Du, N.; Shafran, I.; Narasimhan, K.; and Cao, Y
Springer. Yao, S.; Zhao, J.; Yu, D.; Du, N.; Shafran, I.; Narasimhan, K.; and Cao, Y . 2022. React: Synergizing reasoning and acting in language models. arXiv preprint arXiv:2210.03629. Ye, R.; Zhang, C.; Wang, R.; Xu, S.; Zhang, Y .; et al
2022 arXiv
-
[2021]
In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 7515–7525
Scenegraphfusion: Incremental 3d scene graph predic- tion from rgb-d sequences. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 7515–7525. Xi, Z.; Chen, W.; Guo, X.; He, W.; Ding, Y .; Hong, B.; Zhang, M.; Wang, J.; Jin, S.; Zhou, E.; et ...
2023 arXiv
-
[2022]
Ad- vances in neural information processing systems , 35: 22199– 22213
Large language models are zero-shot reasoners. Ad- vances in neural information processing systems , 35: 22199– 22213. Liang, J.; Huang, W.; Xia, F.; Xu, P.; Hausman, K.; Ichter, B.; Florence, P.; and Zeng, A. 2023. Code as policies: Language model programs for embodied contro...
2023
-
[2023]
arXiv preprint arXiv:2303.16189
Planning with sequence models through iterative en- ergy minimization. arXiv preprint arXiv:2303.16189. Chen, X.; Lin, M.; Schärli, N.; and Zhou, D. 2024. Teach- ing Large Language Models to Self-Debug. In The Twelfth International Conference on Learning Representations . Chen...
2024 arXiv
-
[2024]
arXiv preprint arXiv:2402.00157
Large language models for mathematical reasoning: Progresses and challenges. arXiv preprint arXiv:2402.00157. Ahn, M.; Brohan, A.; Brown, N.; Chebotar, Y .; Cortes, O.; David, B.; Finn, C.; Fu, C.; Gopalakrishnan, K.; Hausman, K.; et al. 2022. Do as i can, not as i say: Ground...
2022 arXiv
Reviewed August 9, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.