{"id":"b7150cb6-e365-42ea-832f-adbb4a49cc08","arxiv_id":"2509.04642","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Maestro jointly searches agent graph topology and node configurations, reporting higher test scores than prompt-only optimizers on HotpotQA, IFBench, and two internal agent tasks.","lead":"Maestro is an optimizer that tunes both the structure of an LLM agent (which modules exist and how they connect) and the settings of each module (prompts, models, tools) at the same time, instead of only tuning prompts in a fixed pipeline. On two public benchmarks it reports gains over existing prompt optimizers with fewer trial runs, and on two internal applications it proposes structural changes such as adding validators or memory state.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Headline margins rest on cited GEPA/MIPROv2 numbers and a re-implemented IFBench agent; without matched re-runs or error bars the claimed gains, especially IFBench config-only +0.17pp, are not yet established.","rationale":"The reader's weakest assumption—that the comparison against GEPA/MIPROv2 is valid despite relying on cited numbers rather than matched re-runs—is the most load-bearing point. The paper's headline is a quantitative superiority claim, and that claim collapses if the underlying measurements are not commensurable. My review of the manuscript found no internal contradiction in the formalization (Sec. 2) or in the high-level Maestro algorithm (Sec. 4); the risk is empirical and external. The IFBench agent is explicitly 're-implemented' rather than directly reused, and the proprietary-algorithm footnote plus absence of code/data makes the provenance of graph edits unverifiable. These are not charges of dishonesty; they are standard reproducibility requirements for a claim that depends on small margins and on a novel search mechanism. The reader's CONDITIONAL verdict is therefore appropriate: the claims are plausible but not yet established, and the concrete re-run test would settle whether the baseline comparability concern actually lands. I see no reason to move the verdict to ACCEPT or REJECT; the evidence supports conditional acceptance pending the stated verification.","tokens_in":22833,"tokens_out":6207,"duration_ms":64075,"concrete_test":"Obtain the released code for GEPA, MIPROv2, and GEPA+Merge and re-run them inside the authors' exact HotpotQA and IFBench pipelines (including the re-implemented IFBench agent), using identical data splits, the same gpt-4.1-mini-2025-04-14 snapshot, the same evaluator prompts, and at least 5 seeds per method. Report mean±std or bootstrap CIs. If the re-run baseline scores differ from the cited values (52.72/55.95 on IFBench, 69.00/65.67 on HotpotQA) by more than the claimed Maestro advantage, or if Maestro's advantage over re-run baselines is not significant at p<0.05, the headline margins are unsupported. In addition, release Maestro's search logs showing that the reported graph edits were proposed by the optimizer before any manual curation.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that Maestro's joint graph+config search beats prompt-only optimizers on IFBench and HotpotQA. For that claim to hold, Maestro's numbers and the cited MIPROv2/GEPA/GEPA+Merge numbers must be measured under identical task instances, evaluator, data splits, LLM snapshot, and agent implementation. The paper asserts it 'follow[s] exactly the evaluation protocol of GEPA' (Secs. 5.1.1, 5.2.1), but it does not re-run those baselines; for IFBench the agent is 're-implemented' (Sec. 5.2.1), not directly reused, and no code, data, or detailed algorithm is released—Sec. 4 explicitly marks technical details as proprietary. Small differences in the re-implementation, evaluator prompts, LLM API version, or rollout accounting could shift the baselines by more than the reported margins. This is especially consequential for the IFBench config-only result (56.12 vs. GEPA+Merge's 55.95, a 0.17pp gap) and the 3.23pp graph+config gain, neither of which has error bars, significance tests, or multiple seeds. Additionally, the graph edits (extract_entities, validate_constraints, branches_done, numeric_compute) are presented only as final outputs, with no search logs or code, so their provenance—whether the G-step actually discovered them rather than manual post-hoc design—cannot be audited. Without matched re-runs and auditable graph-search traces, the abstract's numeric superiority and the 'structural failure modes' narrative are not independently verified.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Maestro, a claimed framework-agnostic optimizer that jointly searches over an agent's computation graph and its node configurations (prompts, models, tools, hyperparameters), under rollout and cost budgets. It formalizes agents as typed stochastic computation graphs and proposes a block-coordinate scheme alternating a C-step (configuration optimization) and a G-step (graph structural edits), guided by both numeric evaluation scores and textual feedback from execution traces. The empirical sections report results on HotpotQA and IFBench using the evaluation protocol of GEPA/MIPROv2, plus two in-house applications (interviewer and RAG agents). The central claims are that Maestro outperforms prompt-only optimizers MIPROv2, GEPA, and GEPA+Merge on the public benchmarks, with further gains from graph edits (e.g., HotpotQA 70.33% config-only vs. 72.33% graph+config; IFBench 56.12% vs. 59.18%), and that structural changes such as entity extraction, a validation module, an explicit state variable, and a numeric compute tool fix failure modes that prompt tuning cannot.","tokens_in":23246,"tokens_out":2117,"duration_ms":22759,"significance":"If substantiated, the contribution would be significant: it extends agent optimization beyond prompt tuning to a structured, budgeted search over agent topology, and it uses textual trace feedback in a principled way. The formal graph/config formulation in Section 2 is a useful unifying language, and the public-benchmark experiments target a question of current interest. However, the empirical validation is not yet at the standard needed to support the headline quantitative claims. The comparisons to GEPA/MIPROv2 rely on previously reported numbers rather than matched re-runs, the IFBench agent is re-implemented rather than reused, no error bars or multiple seeds are provided, and the proprietary nature of the optimizer (Section 4) means the graph-edit results cannot be independently audited. The paper also gives credit-worthy concrete applications and reproducible-looking prompts in the appendix, but the lack of released code or search traces prevents verification of the central 'joint graph+config search discovers structural fixes' narrative.","major_comments":[{"comment":"The headline gains against MIPROv2, GEPA, and GEPA+Merge are not established because the baselines are cited from Agrawal et al. (2025) rather than re-run in the same environment. The paper states 'we follow exactly the evaluation protocol of GEPA' and 'evaluated and reported by Agrawal et al.' but does not re-run those baselines with the same LLM API version, data sampling, evaluator prompts, or agent implementation. For IFBench the agent is explicitly 're-implemented' (Section 5.2.1), so even the underlying agent differs. Since the claimed margins are small—e.g., IFBench config-only 56.12% vs. GEPA+Merge 55.95% (a 0.17pp difference), and HotpotQA config-only 70.33% vs. GEPA 69.00% (1.33pp)—environmental differences could reverse or eliminate the advantage. The paper must either re-run the baselines under identical conditions or report the comparison as preliminary, with explicit caveat","section":"Sections 5.1.1, 5.2.1 and Figures 4–5"},{"comment":"No variance information is reported for any result. All headline numbers are single-point estimates without standard errors, confidence intervals, or multiple seeds. On HotpotQA, the difference between Maestro graph+config (72.33%) and GEPA (69.00%) is close to what could plausibly arise from seed/rollout noise on a 300-example test set. Similarly, the interviewer agent's '2% to 92%' improvement rests on 50 test trajectories generated from one batch of personas, and the RAG agent uses a custom judge without inter-annotator agreement or variance estimates. At minimum, the paper should provide standard errors or bootstrapped CIs for all central claims, and ideally multiple independent runs of the optimizer.","section":"Sections 5.1.2, 5.2.2, 5.3.2, 5.4.2"},{"comment":"The G-step is described only at a high level, with the footnote 'technical details remain proprietary.' The graph edits (extract_entities, validate_constraints, branches_done, numeric_compute) are presented as final outputs, but no search logs, candidate graph trajectories, or ablation of the G-step are provided. This makes it impossible to verify that these edits were discovered by Maestro's search rather than selected post hoc by the authors. Since the central narrative—that joint graph search is necessary for certain failure modes—depends on the provenance of these edits, the paper should release the search trace or, failing that, provide a reproducibility protocol (e.g., a publicly runnable minimal version with a documented random seed and search budget).","section":"Section 4 and Appendix A"}],"minor_comments":[{"comment":"Typo: 'confured' should be 'configured'.","section":"Section 5.3.1, 'LLM' bullet"},{"comment":"The two panels appear visually identical in the provided text; the 'highlighted in blue' distinction is not visible in the reproduction. Please ensure the figure renders the added nodes/edges clearly, and consider labeling the diff explicitly.","section":"Figures 3a and 3b"},{"comment":"The abstract claims Maestro 'surpasses' MIPROv2/GEPA/GEPA+Merge by average margins of 12%, 4.9%, and 4.86%, but the per-benchmark numbers in Sections 5.1.2 and 5.2.2 do not obviously yield these averages, and the exact aggregation is unspecified. Clarify whether the reported percentages are per-benchmark average improvement or pooled across benchmarks.","section":"Abstract and Section 5.2.2"},{"comment":"The formulation uses a structure regularizer Ω(G) and a constraint Ω(G) ≤ τ, but the practical choice of τ and how it interacts with the trust-region radius r_t in Section 4 (both bound graph change) is not discussed. A brief clarification would help.","section":"Section 2.2"},{"comment":"The numeric_compute tool code is presented as a 'suggested' addition, but the appendix does not show whether the tool was actually invoked in the evaluation or only in the optimized agent. Please state explicitly which tools were used in the reported runs.","section":"Appendix A.4"}],"recommendation":"major_revision","confidential_remarks":"The paper is a technical report from a company (RELAI.ai) and explicitly marks algorithm details as proprietary. This is not itself disqualifying, but it makes the empirical claims hard to verify. If the authors can commit to re-running the baselines in a matched setup, reporting error bars, and releasing at least a reproducible artifact (or a detailed search log), the scientific contribution could be solid. Without that, the headline quantitative claims are conditional on the reader trusting the authors' internal implementation. I would encourage the editor to ask for these revisions before considering acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The real contribution here is the framing: treat the agent as a typed computation graph and search over both structure and node configuration under explicit budgets, using textual feedback to guide proposals. That is a genuine extension over GEPA and MIPROv2, which hold the graph fixed, and the block-coordinate C-step/G-step with trust-region edits is a sensible design. The paper also does a decent job of positioning itself against MAAS and other prior work. The optimized prompts in the appendix are detailed and show real engineering effort.\n\nWhat the paper does not yet do is back up its headline margins. The HotpotQA and IFBench numbers are compared against scores cited from Agrawal et al. (2025), not re-runs in the same environment. The IFBench config-only win over GEPA+Merge is 0.17 percentage points—within noise, especially with no error bars or multiple seeds. The internal interviewer and RAG benchmarks are self-generated, and the evaluator is an LLM judge without reported agreement or calibration. The graph edits (extract_entities, validate_constraints, branches_done, numeric_compute) are sensible, but they are presented as final outputs with no search logs or code; the claim that Maestro discovered them cannot be audited. Section 4 says technical details remain proprietary, which compounds the problem. These issues do not falsify the central idea—it is quite plausible that joint search adds value—but they do mean the quantitative claims are conditional until the evidence is opened up.\n\nI would also note that the two application graphs are simple enough that the improvements could plausibly come from prompt changes alone in the config-only condition, but the graph versions do add real state or validation machinery. So the qualitative story holds.\n\nBottom line: this is a promising paper that deserves a serious referee, not a desk reject. The referee should push for code, re-run baselines with variance, and a non-proprietary description of the optimizer. As it stands I would not cite the numeric results, but I would cite the formulation if it appears in a peer-reviewed version. I would bring it to a reading group to discuss the graph-search formulation.","headline":"A plausible and well-formulated idea for joint graph+config agent optimization, but the headline numbers rest on cited baselines and withheld optimizer details.","tokens_in":23746,"tokens_out":1422,"would_cite":false,"duration_ms":17674,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Jointly optimizing agent graphs and configurations beats prompt-only search on IFBench and HotpotQA.","keywords":["LLM agents","agent graph optimization","prompt optimization","configuration search","joint optimization","reflective feedback","HotpotQA","IFBench"],"falsifier":"Re-run the HotpotQA and IFBench comparisons in one harness with identical evaluator prompts, data splits, LLM version (gpt-4.1-mini-2025-04-14), and rollout accounting for both Maestro and the three baselines; if Maestro's margins shrink to noise or Maestro needs more rollouts than GEPA, the paper's central claim would be falsified.","tokens_in":22720,"feed_emoji":"🤖","tokens_out":7558,"duration_ms":67602,"temperature":0.7,"pith_summary":"Maestro is a framework-agnostic optimizer for LLM agents that treats an agent as a typed computation graph and searches both the graph's structure (which modules exist and how they connect) and each node's configuration (prompts, model, tools, control knobs). The paper's central claim is that this joint search beats prompt-only optimizers that keep the architecture fixed: on IFBench and HotpotQA, Maestro surpasses GEPA, MIPROv2, and GEPA+Merge by averages of about 12%, 4.9%, and 4.86%, respectively, and does so with far fewer rollouts. The paper also argues that structural edits—adding an entity-extraction step, a constraints validator, an explicit state variable, or a numeric-computation tool—fix failure modes prompt tuning alone cannot, and that reflective textual feedback from traces steers the search toward those fixes. If right, the result would give practitioners a practical path to improving agent reliability without hand-rewiring graph skeletons.","feed_headline":"Graph edits lift agents up to 12% over prompt-only tuning","feed_subtitle":"Maestro rewrites both structure and prompts, fixing failures prompt tuning cannot.","key_machinery":"The central object is the typed stochastic computation graph G=(V,E): nodes carry capabilities (LLM invocation, retrieval, tool calls, memory modules, validators), edges carry adapters that transform a parent's output into a child's input, and each node has a merge operator combining incoming messages. Maestro's search alternates C-steps (fix the graph, optimize configurations) and G-steps (mutate the graph within an edit-distance trust region), accepting changes under a guarded improvement rule. Reflective non-numeric feedback from execution traces is distilled into targeted proposals, which is what lets structural edits—validators, entity extraction, persistent state, new tools—be discover","core_discovery":"Maestro casts agent design as maximizing expected utility over graph G and configuration C under rollout, cost, and structure budgets, then alternates a C-step (configuration search with the graph fixed) and a G-step (small structural edits, such as adding, removing, or rewiring nodes, with configuration fixed), using textual trace feedback to choose where to edit. On HotpotQA and IFBench, config-only Maestro reaches 70.33% and 56.12%, slightly above GEPA and GEPA+Merge; graph+config reaches 72.33% and 59.18%, with far fewer rollouts. In two applications, graph edits add an explicit state variable (interviewer completions from 2% to 92%) and a numeric tool plus retrieval guidance (RAG score","pith_inferences":["If this joint search generalizes beyond the four case studies, hand-fixing agent graphs and tuning prompts separately looks inefficient; agent development workflows could shift toward budgeted, feedback-driven joint search.","The interviewer agent's improvement suggests many so-called LLM mistakes in long multi-turn dialogues are really missing-state problems, so adding explicit memory registers may be a cheaper fix than fine-tuning—testable by applying the branches_done trick to other multi-turn agents.","A natural stress test is to weaken the evaluator's textual feedback (for example, switch to scalar-only scores); if Maestro's sample-efficiency advantage shrinks, reflective feedback is the load-bearing ingredient, and optimizers that also consume text might close the gap."],"forward_implications":["On the two public benchmarks, Maestro's graph+config mode beats reported GEPA/MIPROv2/GEPA+Merge numbers while spending a small fraction of the rollouts, suggesting prompt-only search leaves a measurable accuracy margin on the table.","The interviewer result (2% to 92% complete rate) and the RAG result (39.1 to 80.4) show structural additions can turn unreliable agents into usable ones, a class of gains inaccessible to prompt tuning.","Since Maestro is framework-agnostic—it registers optimizable configurations and tracks traces without requiring a specific agent implementation—the same optimizer could apply across different agent frameworks and evolving applications.","Joint optimization with explicit rollout and token budgets gives a practical way to trade accuracy against latency and cost rather than optimizing quality alone."],"supporting_citations":[{"why":"Supplies the GEPA baseline, the evaluation protocol, and the reported comparison numbers Maestro exceeds on IFBench and HotpotQA.","marker":"Agrawal et al. (2025)"},{"why":"MIPROv2 is the prompt-optimization baseline Maestro is compared against.","marker":"Opsahl-Ong et al. (2024)"},{"why":"HotpotQA provides the multi-hop question-answering benchmark, data splits, and evaluator used in the experiments.","marker":"Yang et al. (2018)"},{"why":"IFBench provides the instruction-following benchmark and its data splits for the experiments.","marker":"Pyatkin et al. (2025)"},{"why":"DSPy implements the two-hop retrieval agent that Maestro optimizes on HotpotQA.","marker":"Khattab et al. (2023)"},{"why":"MAAS is the prior prompt-plus-topology optimizer whose restricted search space Maestro extends.","marker":"Zhou et al. (2025)"}],"fun_headline_variants":["Joint graph+config tuning beats prompt-only by 12%","Structure + prompts together lift agent scores 12%","Maestro: co-optimize graph and config, beat prompt-only","12% gain: tuning agent graphs and configs jointly","Graph+config search trounces prompt tuning, 12% better"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The headline margins rest on comparing against GEPA, MIPROv2, and GEPA+Merge numbers reported in GEPA's own paper under GEPA's exact protocol, without re-running those baselines in Maestro's environment; if slight differences in evaluator prompts, LLM versions, or data sampling make the baselines non-comparable, the gains would not be established.","fun_headline_variants_meta":{"raw":{"variants":["Joint graph+config tuning beats prompt-only by 12%","Structure + prompts together lift agent scores 12%","Maestro: co-optimize graph and config, beat prompt-only","12% gain: tuning agent graphs and configs jointly","Graph+config search trounces prompt tuning, 12% better"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000153,"raw_usage":{"total_tokens":1055,"prompt_tokens":765,"completion_tokens":290,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":509,"completion_tokens_details":{"reasoning_tokens":213}},"tokens_in":509,"tokens_out":290,"duration_ms":3043,"temperature":1.0,"reasoning_tokens":213,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T05:55:47.828798+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the HotpotQA and IFBench comparisons in one harness with identical evaluator prompts, data splits, LLM version (gpt-4.1-mini-2025-04-14), and rollout accounting for both Maestro and the three baselines; if Maestro's margins shrink to noise or Maestro needs more rollouts than GEPA, the paper's central claim would be falsified.","supporting_citations":[],"review_version":1}