{"id":"c389c09d-34cf-4dbb-b7f7-b70d097712c4","arxiv_id":"2604.23626","paper_version":1,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":7.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":0,"one_line_summary":"GraphPlanner augments multi-agent LLM routing with a heterogeneous graph memory and RL-optimized MDP workflow generation, delivering up to 9.3% higher accuracy and over 99% lower GPU cost than prior routers while supporting zero-shot generalization.","lead":"GraphPlanner is a router that builds workflows for teams of different AI models using a graph to remember past interactions and decisions. It claims to boost accuracy on complex tasks while slashing computing costs and working on new problems without retraining.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.3","headline":"RL policy on GARNet may overfit training tasks, undermining robust generalization to unseen tasks/LLMs","rationale":"The load-bearing risk matches the reader's weakest assumption exactly; because the review was abstract-only, the same unverified RL+graph generalization issue remains the primary soft spot even after full-text consideration.","tokens_in":1824,"tokens_out":309,"duration_ms":73083,"concrete_test":"Retrain GraphPlanner after ablating GARNet (replace with flat history buffer) and evaluate zero-shot on held-out task categories disjoint from the original 14; if accuracy gain over baselines falls below 4% or GPU savings shrink substantially, the graph memory is not enabling the claimed generalization.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim requires that formulating workflow generation as an MDP (selecting LLM backbone + role at each step) and optimizing via RL on the heterogeneous GARNet graph memory produces routing decisions that generalize via inductive/transductive inference. This is the least secure link because RL policies on graph-structured states frequently overfit to training query distributions and interaction patterns; if GARNet fails to encode useful long-term agent-response memories beyond memorization, the reported 9.3% accuracy lift and GPU reduction (186.26 GiB to 1.04 GiB) plus zero-shot claims would not hold. The abstract asserts effective memory leverage but supplies no ablations, regularization details, or policy analysis to confirm the graph prevents this failure mode.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.3","summary":"The manuscript proposes GraphPlanner, a heterogeneous graph memory-augmented agentic router for multi-agent LLMs. It formulates workflow generation as an MDP in which the policy at each step selects both an LLM backbone and an agent role (Planner, Executor, or Summarizer). A heterogeneous graph GARNet encodes interaction memories among queries, agents, and responses and augments the state representation; the full pipeline is trained with reinforcement learning to jointly optimize task performance and efficiency. Experiments across 14 LLM tasks are reported to demonstrate up to 9.3% accuracy gains over strong single- and multi-round routers, a reduction in GPU cost from 186.26 GiB to 1.04 GiB, robust zero-shot generalization to unseen tasks and LLMs, and effective use of historical memories for both inductive and transductive inference. Code is released.","tokens_in":1980,"tokens_out":667,"duration_ms":56766,"significance":"If the empirical results hold after addressing the evidentiary gaps, the work would constitute a useful extension of LLM routing into realistic agentic, multi-round settings by combining graph memory with RL-based workflow planning. The public code release is a clear strength that supports reproducibility and further research.","major_comments":[{"comment":"Experiments section: The central performance claims (up to 9.3% accuracy improvement and GPU cost reduction from 186.26 GiB to 1.04 GiB) are presented without error bars, detailed descriptions of the 14 tasks, baseline implementations, or ablation studies that isolate the contribution of the GARNet graph memory versus the MDP/RL formulation alone. This omission makes it impossible to verify whether the graph memory is load-bearing for the reported gains.","section":"Experiments section"},{"comment":"Generalization claims (zero-shot to unseen tasks/LLMs, inductive/transductive inference): No policy analysis, overfitting diagnostics, regularization details, or comparisons against non-graph RL baselines are provided. RL policies on graph-structured states are known to overfit training query distributions; without such evidence the generalization results cannot be taken as confirmed.","section":"Experiments / Generalization subsection"},{"comment":"Method section (GARNet and state representation): The precise update and retrieval mechanisms by which GARNet incorporates historical memories into the MDP state are described only at a high level. This leaves open whether the memory augmentation introduces circularity or simply memorizes training patterns rather than enabling robust inference.","section":"Method section"}],"minor_comments":[{"comment":"The abstract introduces several acronyms (MDP, GARNet, LLM) without expansion on first use; a brief parenthetical definition would improve readability.","section":"Abstract"},{"comment":"Notation for the heterogeneous graph and its node/edge types is introduced without a compact mathematical definition or diagram reference, complicating replication of the state representation.","section":"Method section"}],"recommendation":"major_revision","confidential_remarks":"The abstract presents unusually strong numerical claims with essentially no methodological or statistical detail; this is atypical for a submission and warrants checking whether the full experimental reporting meets the journal's standards for reproducibility."},"author_rebuttal":{"model":"grok-4.3","summary":"We thank the referee for the constructive and detailed feedback. We agree that the manuscript would benefit from greater experimental rigor and methodological precision to better substantiate the claims. We address each major comment below and will incorporate the suggested revisions in the next version of the manuscript.","responses":[{"response":"We agree that the current presentation lacks sufficient statistical rigor and component isolation. In the revised manuscript we will add error bars computed over at least five random seeds for all accuracy and cost metrics. Detailed descriptions of the 14 tasks will be moved to an appendix. Baseline implementations will be clarified with exact hyper-parameters and references to the released code. We will also include new ablation studies comparing the full model against (i) an MDP/RL variant without GARNet and (ii) a non-RL variant with fixed workflows, thereby isolating the contribution of the heterogeneous graph memory.","revision_made":"yes","referee_comment":"[Experiments section] Experiments section: The central performance claims (up to 9.3% accuracy improvement and GPU cost reduction from 186.26 GiB to 1.04 GiB) are presented without error bars, detailed descriptions of the 14 tasks, baseline implementations, or ablation studies that isolate the contribution of the GARNet graph memory versus the MDP/RL formulation alone. This omission makes it impossible to verify whether the graph memory is load-bearing for the reported gains."},{"response":"We acknowledge the risk of overfitting in graph-augmented RL policies. The revised version will include (i) policy visualizations contrasting behavior on training versus held-out queries, (ii) training/validation performance curves and explicit regularization details (dropout in GARNet layers and entropy regularization in the RL objective), and (iii) direct comparisons against non-graph RL baselines. These additions will provide the requested diagnostics and strengthen the zero-shot, inductive, and transductive generalization claims.","revision_made":"yes","referee_comment":"[Experiments / Generalization subsection] Generalization claims (zero-shot to unseen tasks/LLMs, inductive/transductive inference): No policy analysis, overfitting diagnostics, regularization details, or comparisons against non-graph RL baselines are provided. RL policies on graph-structured states are known to overfit training query distributions; without such evidence the generalization results cannot be taken as confirmed."},{"response":"We will expand the Method section with formal definitions of the update and retrieval operations, including pseudocode for memory insertion and similarity-based retrieval at each MDP step. We will explicitly clarify that memory updates occur after inference and that retrieval uses embedding similarity rather than exact pattern matching, thereby avoiding circularity. Additional discussion will explain how the joint RL objective favors generalizable routing policies over memorization, supported by the new generalization diagnostics.","revision_made":"yes","referee_comment":"[Method section] Method section (GARNet and state representation): The precise update and retrieval mechanisms by which GARNet incorporates historical memories into the MDP state are described only at a high level. This leaves open whether the memory augmentation introduces circularity or simply memorizes training patterns rather than enabling robust inference."}],"tokens_in":1601,"tokens_out":660,"duration_ms":42428,"standing_objections":[]},"desk_editor":{"model":"grok-4.3","letter":"The new piece is treating workflow generation as an MDP that selects both LLM backbone and agent role (planner, executor, summarizer) at each step, then feeding a heterogeneous graph called GARNet with past query-agent-response triples into the state for RL optimization. That setup lets the router pull historical memories for both inductive and transductive decisions, which is a clean way to extend single-round routers into multi-round agentic settings without hand-crafted prompts for every task.","headline":"GraphPlanner combines a heterogeneous graph memory with an MDP policy for picking models and roles in multi-agent workflows, delivering reported cost cuts but resting on unablated generalization claims.","tokens_in":2476,"tokens_out":171,"would_cite":false,"duration_ms":47830,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.3","headline":"GraphPlanner uses a heterogeneous graph memory and reinforcement learning to generate adaptive routing workflows for multi-agent LLMs.","keywords":["LLM routing","multi-agent LLMs","graph memory","reinforcement learning","agentic workflows","Markov Decision Process","workflow generation","heterogeneous graphs"],"falsifier":"An evaluation on a new set of tasks or LLMs where GraphPlanner shows no accuracy improvement over strong single-round and multi-round baselines, or where the claimed GPU cost reduction from 186.26 GiB to 1.04 GiB does not occur.","tokens_in":2727,"feed_emoji":"🤖","tokens_out":751,"duration_ms":46969,"temperature":0.7,"pith_summary":"The paper introduces GraphPlanner to address routing challenges in agentic settings where multiple LLMs must cooperate through planning, execution, and summarization steps. It models workflow creation for each query as a Markov Decision Process, with actions that pick both an LLM and a role such as Planner, Executor, or Summarizer. A heterogeneous graph called GARNet stores memories of past query-agent-response interactions, which are folded into the decision state so reinforcement learning can optimize the full pipeline for accuracy and efficiency. A sympathetic reader would care because current routers handle only simple single-round or multi-round cases and become impractical for realistic multi-agent applications that need memory and generalization.","feed_headline":"Graph memory router improves multi-agent LLM accuracy and efficiency","feed_subtitle":"By casting workflow generation as an MDP and storing agent interactions in GARNet, the system generalizes across tasks while using far less ","key_machinery":"The GARNet heterogeneous graph that captures interaction memories among queries, agents, and responses, which augments the state representation in the MDP formulation for joint RL optimization of routing decisions.","core_discovery":"GraphPlanner generates routing workflows for each query by formulating the process as a Markov Decision Process where at each step the system selects both an LLM backbone and an agent role from Planner, Executor, and Summarizer. The state representation is enriched with historical and workflow memories drawn from a heterogeneous graph GARNet that records interactions among queries, agents, and responses. The full pipeline is trained end-to-end with reinforcement learning to improve task-specific performance while lowering computational cost, supporting both inductive and transductive inference on unseen tasks and models.","pith_inferences":["The MDP-plus-graph formulation could be applied to other sequential routing or planning problems where past interaction data improves future decisions.","If the graph memory continues to scale, it may allow routers to adapt to new LLMs without full retraining by simply extending the stored interaction records.","Removing the graph component in an ablation would likely eliminate the reported gains in generalization and cost reduction."],"forward_implications":["GraphPlanner outperforms strong single-round and multi-round routers, improving accuracy by up to 9.3% across 14 diverse LLM tasks.","It reduces GPU cost from 186.26 GiB to 1.04 GiB while maintaining or increasing performance.","It generalizes robustly to unseen tasks and LLMs, exhibiting strong zero-shot capabilities.","It effectively leverages historical memories to support both inductive and transductive inference for more adaptive routing."],"fun_headline_variants":["GraphPlanner uses MDP to select LLM backbones and agent roles","Heterogeneous graph GARNet captures memories among agents and queries","GraphPlanner jointly optimizes task performance and computational cost","The system supports zero-shot routing on unseen tasks and models"],"cache_read_input_tokens":64,"weakest_assumption_plain":"That formulating workflow generation as an MDP and optimizing it with RL on the GARNet graph memory will reliably produce superior routing decisions without the learned policy overfitting to the training tasks or the graph failing to capture useful long-term interaction patterns.","fun_headline_variants_meta":{"raw":{"variants":["GraphPlanner uses MDP to select LLM backbones and agent roles","Heterogeneous graph GARNet captures memories among agents and queries","GraphPlanner jointly optimizes task performance and computational cost","The system supports zero-shot routing on unseen tasks and models"]},"model":"grok-4.3","cost_usd":0.011201,"raw_usage":{"total_tokens":4973,"prompt_tokens":770,"num_sources_used":0,"completion_tokens":64,"cost_in_usd_ticks":112012000,"prompt_tokens_details":{"text_tokens":770,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":4139,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":770,"tokens_out":64,"duration_ms":50218,"temperature":1.0,"reasoning_tokens":4139,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-05-08T06:15:08.908710+00:00","model_set":{"reader":"grok-4.3"},"falsifier":"An evaluation on a new set of tasks or LLMs where GraphPlanner shows no accuracy improvement over strong single-round and multi-round baselines, or where the claimed GPU cost reduction from 186.26 GiB to 1.04 GiB does not occur.","supporting_citations":[],"review_version":1}