{"id":"22ff7eba-1d97-47c0-910b-5cc0d3950431","arxiv_id":"2603.02630","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"MASPOB combines a GNN surrogate, LinUCB-style uncertainty, and coordinate ascent to optimize prompts in fixed-topology multi-agent LLM systems, beating AFlow and MIPRO on average across six benchmarks.","lead":"MASPOB is an optimizer that picks better prompts for each agent in a fixed multi-agent LLM workflow, using a graph neural network to model how agents depend on each other and a bandit-style uncertainty bonus to explore within a 50-evaluation budget. On six QA, code, and math benchmarks it reports higher average accuracy than AFlow and MIPRO, though the 'best on every benchmark' claim is weakened by a tie on MBPP.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"MASPOB vs. MIPRO does not hold prompt-domain construction fixed; its 1.71-pt average gain could come from GPT-4o-mini candidate generation rather than the bandit/GNN optimizer.","rationale":"The reader's weakest assumption concerned the candidate prompt domain, and the paper's own limitation statement identifies the same dependency. I agree that the candidate domain is a key vulnerability. However, the more load-bearing issue is not just that the optimizer is limited to a pre-generated set, but that the headline comparison against MIPRO does not hold that set fixed across methods. The missing symmetric control (MIPRO on MASPOB's domain) is the single experiment that would settle whether the claimed gains come from MASPOB's search strategy or from the richer/differently constructed candidate pool. Table 5 partially addresses domain robustness for MASPOB itself, but it does not isolate the optimizer-vs-domain attribution for the main pairwise comparison. This does not overturn the paper's otherwise reasonable empirical protocol; it makes the central claim conditional on a specific missing ablation. The reader already returned CONDITIONAL, and this concern reinforces that verdict rather than moving it to rejection.","tokens_in":32760,"tokens_out":10066,"duration_ms":101629,"concrete_test":"Run MIPRO on exactly the MASPOB candidate domains (the 20 GPT-4o-mini style-controlled variants per agent) on all six benchmarks, using the same fixed AFlow workflows, the same 50-evaluation budget, and the same three-run test protocol. Compare MIPRO's average test score to MASPOB's reported 80.58. If MIPRO matches or exceeds MASPOB on this shared domain, the central claim of optimizer superiority fails; if MIPRO remains clearly below (e.g., more than 1.5 points), the concern is resolved. For completeness, report both directions of the domain swap, including MASPOB on MIPRO's domain, rather than only the one direction in Table 5.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim is that MASPOB outperforms strong baselines under the same 50-evaluation budget. For the closest baseline, MIPRO, the comparison is confounded with prompt-domain construction. MASPOB builds its candidate pool with 20 GPT-4o-mini style-controlled paraphrases per agent (Appendix A.3), while MIPRO constructs its own domain via data-aware, program-aware, fewshot-aware, and tip-aware strategies. Section 4 concedes that \"the quality and diversity of candidate prompts can still affect absolute performance, since the optimizer can only select from the provided prompt domain.\" Table 5 only swaps MIPRO's domain into MASPOB and shows similar scores; it never runs MIPRO on MASPOB's domain. Therefore the reported 1.71 percentage-point average advantage over MIPRO could be partly or wholly due to a better candidate pool rather than to the GNN/UCB/coordinate-ascent selection mechanism. The paper's own ablation attributes gains to the optimizer components, but that attribution is valid only if the candidate domain is controlled across optimizers. Without this control, the \"consistently outperforms\" claim is not fully supported.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes MASPOB, a prompt optimizer for multi-agent systems (MAS) with a fixed workflow topology. The method combines a GAT-based surrogate, a LinUCB-style exploration bonus, and coordinate ascent to search a discrete Cartesian product of per-agent prompt variants under a tight evaluation budget (50 end-to-end executions). The authors evaluate on six benchmarks spanning QA, code generation, and mathematical reasoning, comparing against single-agent baselines (IO, CoT, ReAct, PromptBreeder, Instinct) and multi-agent methods (AFlow, MIPRO). They report an average score of 80.58% and claim consistent state-of-the-art performance. Ablations investigate the GNN, uncertainty estimation, warm-up, embedding model, coordinate ascent, and exploration coefficient.","tokens_in":33133,"tokens_out":5605,"duration_ms":55527,"significance":"If the results hold, MASPOB addresses a practically important and under-studied problem: sample-efficient prompt optimization for already-validated, frozen MAS workflows. The combination of a topology-aware GNN surrogate with a linear UCB bonus is well motivated, and the paper provides a fairly extensive experimental suite with a fixed 50-evaluation budget, ablations on several design components, robustness checks across LLMs and prompt domains, and a complexity comparison against exhaustive search. The public code link and detailed hyperparameters (Appendix A.3) support reproducibility. The main contribution is empirical rather than theoretical: no regret or convergence guarantees are given, but the algorithmic structure is sensible and the experimental setup is largely careful. The principal weakness is that the headline comparison against the closest prompt-optimization baseline, MIPRO, is not fully controlled with respect to how the candidate prompt pool is constructed, and a stated claim of 'best on every benchmark' is contradicted by a tie in Table 1.","major_comments":[{"comment":"The sentence 'MASPOB achieves the best result on every benchmark' is not supported by Table 1. On MBPP, MASPOB and MIPRO both report 80.65, i.e., a tie, and the table's bolding gives MASPOB sole credit. The abstract's 'consistently outperforming' is likewise too strong. Please revise to 'matches or outperforms' or otherwise qualify the claim, and correct the bolding.","section":"§4.2, Table 1, Abstract"},{"comment":"The comparison between MASPOB and MIPRO is confounded with prompt-domain construction. MASPOB builds its candidate pool via 20 GPT-4o-mini style-controlled paraphrases per agent (Appendix A.3), while MIPRO uses its own data/program/fewshot/tip-aware strategies. Table 5 only swaps MIPRO's domain into MASPOB, showing similar scores; it never runs MIPRO on MASPOB's domain. Section 4 also concedes that 'the quality and diversity of candidate prompts can still affect absolute performance.' Therefore the reported 1.71-point average advantage over MIPRO could be partly or wholly due to the candidate pool rather than the GNN/UCB/coordinate-ascent selection. Please run MIPRO on the MASPOB candidate pool (or otherwise hold the candidate domain fixed across optimizers) and report both directions before claiming that the gains 'mainly come from topology-aware contextual-bandit optimization.'","section":"§4.2, Table 5, Appendix A.3"},{"comment":"The paper identifies MAPRO (Zhang et al., 2025c) as 'the closest prior work to ours' and as a principled multi-agent prompt optimizer, yet MAPRO is never included as a baseline. For a state-of-the-art claim, omitting the closest competitor is a significant gap. Please add MAPRO to the main comparisons if its code/API allows, or provide a concrete reason (e.g., incompatibility with the fixed-budget protocol) why it cannot be included.","section":"§6 Related Work, Tables 1-4"}],"minor_comments":[{"comment":"The caption has a typo: 'We report mean accuracy (standard deviation over three runs' is missing the '±' and an opening bracket. Also, Table 5 reports only DROP and MATH, while the text in §4.2 says 'as shown in Table 5' without noting the limited coverage; please state the scope explicitly.","section":"Table 5 caption"},{"comment":"The hyperparameter table lists a 'Fisher matrix update coefficient' of 10, but Algorithm 1 and Eq. (10) update the information matrix as M ← M + Φ(c)Φ(c)⊤ with no coefficient. Either the algorithm description is missing a scaling factor or the table entry is unused. Please clarify.","section":"§3.2, Algorithm 1, Table 8"},{"comment":"MATH is referred to as 'MATHlv5*' in the metrics paragraph, which is inconsistent with the dataset name 'MATH' elsewhere. Please unify terminology.","section":"§4.1, Metrics"},{"comment":"The convergence figure reports test accuracy at checkpoints every 5 rounds and validation as a binned average, but the caption does not define how the test checkpoints are averaged (the text says 'evaluated and averaged over three runs'). Please state whether the three runs are the three final test repetitions or separate optimization runs.","section":"§4.2, Figure 3"},{"comment":"The 12.02% average improvement over IO is emphasized as a headline result, but IO is a single-LLM-call baseline while MASPOB uses a multi-agent workflow with multiple LLM calls. The evaluation budget is matched in number of full-workflow executions, not in inference cost. Please add a sentence clarifying this distinction so readers do not interpret the gain as being achieved at equal total LLM inference cost.","section":"§4.2, Table 1"}],"recommendation":"major_revision","confidential_remarks":"The core methodological idea is sound and the experiments are substantial, but the central comparison against MIPRO needs a controlled prompt-domain experiment, and the 'best on every benchmark' claim must be corrected. The missing MAPRO baseline should also be addressed. These are fixable within the manuscript's scope, so I recommend major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a solid empirical systems paper with a genuinely new combination—GNN-based topology-aware surrogate, LinUCB uncertainty, coordinate ascent—for prompt optimization in frozen-topology multi-agent systems. It deserves referee time, but the authors oversell it, and the comparison against MIPRO is not as clean as the prose suggests.\n\nWhat is new: the explicit treatment of workflow topology as a graph prior for prompt search. MAPRO uses belief propagation; MASPOB uses a GAT to encode prompt embeddings and inter-agent dependencies and a LinUCB-style bonus for exploration. That combination is not in the cited literature. The experiments span six benchmarks, use a fixed 50-evaluation budget for the main comparison, and include useful ablations: removing the GNN costs ~2.3 points, coordinate ascent matches global search with 99% runtime savings, and warm-up and exploration coefficient are swept. Code is public.\n\nWhere it gets soft:\n\n- 'Best result on every benchmark' is false: MASPOB ties MIPRO on MBPP at 80.65. Minor, but it makes the abstract and Section 4 overstate.\n- The 12.02% improvement over IO is a percentage-point claim against a single-agent baseline that is not inference-cost-matched. It is not an apples-to-apples efficiency claim.\n- The bigger issue is that the candidate prompt domain is not controlled between MASPOB and MIPRO. MASPOB's pool is 20 GPT-4o-mini style-controlled paraphrases per agent; MIPRO constructs its own domain with data/program/fewshot/tip-aware strategies. Table 5 swaps MIPRO's domain into MASPOB and shows similar scores, which is useful robustness evidence, but that only shows MASPOB is robust to the domain it gets, not that MASPOB would beat MIPRO if both started from the same pool. The paper never runs MIPRO on MASPOB's domain. Given the paper's own admission that 'the optimizer can only select from the provided prompt domain,' the 1.71-point average edge over MIPRO could be partly or wholly a property of the candidate set rather than the bandit/GNN selection mechanism. That needs a control experiment.\n\nNo circularity or fabricated results; the ablation story is consistent with the method's intended mechanism. The writing is clear.\n\nWorth sending to peer review, but the authors should fix the overclaims and add the missing domain-control run before acceptance.","headline":"Useful new combination of GNN surrogate and bandit search for prompt optimization in frozen-topology MAS, but the headline overclaims and the comparison against MIPRO is confounded by candidate-prompt domain.","tokens_in":33552,"tokens_out":2553,"would_cite":true,"duration_ms":24533,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["68T07","68T05"],"pacs":[],"model":"deepseek-v4-flash","headline":"Multi-agent system prompts can be optimized within 50 evaluations by a bandit search whose surrogate reads the workflow's graph, and this beats existing single-agent and multi-agent prompt optimizers across six LLM benchmarks.","keywords":["prompt optimization","multi-agent systems","large language models","contextual bandits","graph neural networks","upper confidence bound","coordinate ascent","sample efficiency"],"falsifier":"Run MASPOB on the same budgets but with a deliberately degraded candidate pool (e.g., 20 near-duplicate paraphrases of one prompt) and compare to random selection from that pool: if random search matches or beats MASPOB, the bandit/GNN machinery is not the source of the reported gains. The complementary test is to give the same 50-evaluation budget to a simple evolutionary search over an identical candidate pool: if the simple baseline matches MASPOB's scores, the topology-aware surrogate and UCB exploration contribute nothing beyond ordinary search.","tokens_in":32677,"feed_emoji":"🤖","tokens_out":6568,"duration_ms":58474,"temperature":0.7,"pith_summary":"Many deployed multi-agent LLM systems cannot have their workflows restructured, because the pipelines are compliance- or expert-vetted; the only lever left is the text of each agent's prompt. MASPOB claims that this lever is enough: it treats prompt search as a budgeted black-box optimization and finds well-coordinated prompt combinations in only 50 end-to-end runs. Its core claim is that the search works by modeling the workflow's topology—using a graph neural network surrogate—so that changing one agent's prompt is evaluated with respect to how it changes what downstream agents receive, while a bandit-style upper-confidence-bound rule decides which combinations to try next. Across six benchmarks (question answering, code generation, and mathematical reasoning) the paper reports that MASPOB outperforms strong single- and multi-agent baselines under the same evaluation budget, averaging 80.58%. The paper is explicit that this result is conditional on the candidate prompt pool: the optimizer selects among pre-generated variants, so it cannot recover a prompt that was never drafted.","feed_headline":"Graph-aware bandit tunes multi-agent prompts in 50 trials","feed_subtitle":"It beats single- and multi-agent baselines on six QA, code, and math benchmarks","key_machinery":"Three components carry the argument. (1) A graph neural network (GNN) surrogate: the workflow is a DAG; each agent is a node whose feature is the embedding of its current prompt, and attention-based message passing lets the surrogate predict how a prompt change ripples downstream. (2) A linear upper-confidence-bound (UCB) rule: an information matrix accumulates the embeddings of evaluated combinations; the term sqrt(φ(c)ᵀ M⁻¹ φ(c)) estimates uncertainty, and the acquisition score adds this to the GNN's predicted score, balancing exploitation and exploration. (3) Coordinate ascent: starting from the incumbent best combination, each agent's prompt is greedily replaced with the one maximizing U","core_discovery":"The paper's central discovery claim is that the bottleneck in optimizing fixed-topology multi-agent systems is not the prompt search per se, but the way the search handles inter-agent coupling—and that this coupling can be modeled with a graph. MASPOB represents each agent as a node in the workflow DAG, feeds the candidate prompts' embeddings through a graph-neural-network surrogate to predict system performance, and adds a linear upper-confidence-bound bonus that grows with how unexplored a prompt combination is in embedding space. A coordinate-ascent loop then updates one agent's prompt at a time against the UCB score, cutting the per-round search from exponential to linear in the number o","pith_inferences":["Because MASPOB only selects among pre-written candidates, its ceiling is set by the prompt generator; a natural next step is to let the bandit's uncertainty signal trigger the drafting of new variants in unexplored regions, converting selection into closed-loop generation.","The reported scaling suggests the GNN surrogate's advantage should grow with workflow size and coupling: a testable prediction is that on workflows with more agents or with feedback edges (currently excluded by the DAG assumption), a topology-aware surrogate will separate from structure-blind baselines by a larger margin than the roughly 2.3 points seen here.","The near-tie between coordinate ascent and global search suggests the UCB bonus itself may be supplying the global exploration that makes coordinate-wise greedy updates safe; ablating the exploration coefficient or removing the bonus on a fixed budget would test whether the safety comes from the bonus or from the smoothness of the performance landscape.","For practitioners in regulated settings, the practical implication is that prompt tuning can substitute for workflow restructuring up to a point: the paper's protocol preserves the audited topology and still delivers gains, which aligns with deployment constraints where re-validation of the workflow is expensive or forbidden."],"forward_implications":["With a budget of 50 end-to-end evaluations, MASPOB improves average test accuracy by about 12 percentage points over plain prompting and about 2 points over the strongest multi-agent prompt baseline.","The improvements appear across all six benchmarks—QA, code generation, and math—suggesting the benefit comes from coordination rather than task-specific prompt content.","Removing the graph surrogate costs about 2.3 average points, indicating that topology-aware modeling is a measurable source of the gain, not a cosmetic addition.","Coordinate ascent matches exhaustive search within roughly 0.3–0.5 points but runs 98–99.8% faster, so the search for good prompt combinations is computationally practical.","The benefit transfers to a different backbone LLM and to an alternative, independently generated candidate pool, which the paper interprets as evidence that the gains come from the optimization procedure rather than from a single model or a single prompt-domain recipe."],"fun_headline_variants":["Bandit+GNN prompt search outdoes prior multi-agent baselines","Sample-efficient prompt optimization: MASPOB uses GNNs and UCB","Graph neural bandit tunes agent prompts in fewer trials","Multi-agent prompt optimization via graph-aware bandits"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"MASPOB can only pick from fixed, pre-generated candidate prompts per agent, so if the best prompt for an agent was never drafted, the search cannot find it and all reported gains are bounded by the quality and diversity of that initial candidate pool (a limitation the paper states explicitly in Section 4 and Appendix A.3).","fun_headline_variants_meta":{"raw":{"variants":["Bandit+GNN prompt search outdoes prior multi-agent baselines","Sample-efficient prompt optimization: MASPOB uses GNNs and UCB","Graph neural bandit tunes agent prompts in fewer trials","Multi-agent prompt optimization via graph-aware bandits"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000448,"raw_usage":{"total_tokens":2106,"prompt_tokens":762,"completion_tokens":1344,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":506,"completion_tokens_details":{"reasoning_tokens":1274}},"tokens_in":506,"tokens_out":1344,"duration_ms":9486,"temperature":1.0,"reasoning_tokens":1274,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-02T19:17:37.389087+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run MASPOB on the same budgets but with a deliberately degraded candidate pool (e.g., 20 near-duplicate paraphrases of one prompt) and compare to random selection from that pool: if random search matches or beats MASPOB, the bandit/GNN machinery is not the source of the reported gains. The complementary test is to give the same 50-evaluation budget to a simple evolutionary search over an identical candidate pool: if the simple baseline matches MASPOB's scores, the topology-aware surrogate and UCB exploration contribute nothing beyond ordinary search.","supporting_citations":[],"review_version":1}