{"id":"e085c889-9e31-4c13-b119-fd5120470da9","arxiv_id":"2607.07548","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":3,"one_line_summary":"Scaling the delegation backbone in hierarchical search agents improves EM by ~11 points while scaling the executor moves EM by only ~2.6 points, and a 1.7B SFT executor matches a frontier sub-agent at 37% fewer tokens.","lead":"This paper shows that in multi-agent search systems, the model that decomposes questions matters far more than the model that retrieves answers. A smart business person might read this to learn how to build cheaper AI search agents: use one big model to plan and a tiny one to search.","discovery_kind":"unclear","skeptic_critique":{"model":"glm-5.2","headline":"The delegation-vs-execution asymmetry is measured at asymmetric operating points: delegation sweep fixes executor at its weakest (1.7B), execution sweep fixes backbone at its strongest (DeepSeek-V4-Flash). This design mechanically inflates Δ_D and deflates Δ_E.","rationale":"The reader identified the answerer configuration (π_A fixed, restricted input) as the weakest assumption. This is a valid secondary concern — a weaker answerer or one with access to raw passages might make execution quality more salient. However, the more fundamental issue is the sweep design itself. Even with the current answerer, the asymmetry could reverse or narrow if the cross-sections were chosen at symmetric operating points. The reader's concern is about a boundary condition (what if the answerer were different?); my concern is about the internal validity of the comparison itself (are the two slopes comparable?). Both point to the same conclusion: the bottleneck attribution is conditional, not universal. The paper's practical recommendation — pair a strong backbone with a compact executor — is probably sound for deployment, since you would indeed use a strong backbone in practice. But the general scientific claim that 'decomposition is the capability bottleneck' overstates what the evidence supports. The evidence shows that delegation matters more than execution *when execution is already weak* and *when delegation is already strong* — which is a tautological artifact of testing each axis at the other's extreme. The SFT executor result (Table 5) is a nice independent data point: the 1.7B-SFT matches DeepSeek-V4-Flash (41.81 vs 41.77 EM), but this is tested only under strong backbones (DeepSeek-V4-Flash and GLM-5.1), so it confirms that execution doesn't matter much *when decomposition is good* — which is the same conditional. The data contamination concern (DeepSeek-V4-Flash 36.65 EM without retrieval, App. A) is worth noting but less load-bearing: the backbone doesn't write answers, and the answerer (Qwen3-32B) was chosen to avoid contamination. Contamination could inflate decomposition quality if DeepSeek knows answers and decomposes better as a result, but this is a weaker confound than the sweep asymmetry. Verdict should remain CONDITIONAL; the reader's conditions are valid but the primary condition should be the sweep design, not just the answerer configuration.","tokens_in":16606,"tokens_out":3739,"duration_ms":350707,"concrete_test":"Run two additional cross-sections: (1) delegation sweep with executor fixed at DeepSeek-V4-Flash instead of Qwen3-1.7B — measure Δ_D at strong execution; (2) execution sweep with backbone fixed at Qwen3-1.7B instead of DeepSeek-V4-Flash — measure Δ_E at weak delegation. If Δ_D at strong executor remains > 3× Δ_E at weak backbone, the asymmetry is robust. If the ratio narrows below 2×, the bottleneck attribution is conditional on the sweep design and the headline claim should be qualified.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim — that decomposition is the capability bottleneck — rests on comparing two marginal effects: Δ_D ≈ 11 EM (Table 2, backbone varied with executor fixed at Qwen3-1.7B) versus Δ_E ≈ 2.6 EM (Table 3, executor varied with backbone fixed at DeepSeek-V4-Flash). These two sweeps are measured at different operating points of the held-fixed variable. The delegation sweep starts from the global minimum of the grid (1.7B backbone + 1.7B executor) and measures how much improving delegation helps when execution is at its worst. The execution sweep starts from a near-optimal delegation point (DeepSeek-V4-Flash backbone) and measures how much improving execution helps when decomposition is already near-ceiling. This is a classic interaction-effect problem: the two slopes are measured on different cross-sections of the (c_D, c_E) surface, and the design choice — weakest executor for one sweep, strongest backbone for the other — maximizes Δ_D while minimizing Δ_E. The paper's own SFT analysis (Sec. 3.4, Table 4) shows the key executor capability gap is multi-search correction (1.7B does single-search 99.83% of the time vs. 40.31% multi-search for the teacher). A stronger executor could plausibly compensate for a weak backbone's poor sub-queries by reformulating and iterating — but this compensation can only manifest when the backbone is weak, exactly the regime the execution sweep does not test. Conversely, a strong executor might compress the delegation sweep by compensating for poor decomposition. The paper does not populate the full grid or justify why these specific cross-sections were chosen. Without the off-diagonal cross-sections (delegation sweep at strong executor; execution sweep at weak backbone), the 4× asymmetry ratio could be an artifact of the sweep design rather than a property of the system.","agreement_with_reader":"partial"},"referee_report":{"model":"glm-5.2","summary":"This paper studies how model capacity should be distributed across roles in hierarchical (multi-agent) search agents. The authors factorize the search task into three roles—delegation (task decomposition by a main agent), execution (retrieval and evidence extraction by sub-agents), and answer generation (held fixed as a confound control). Through controlled capacity sweeps on five multi-hop QA benchmarks, they find that (1) role factorization consistently outperforms a single-agent baseline, (2) scaling the delegation backbone improves EM by ~11 points whereas scaling the execution sub-agent moves EM by only ~2.6 points, identifying decomposition as the capability bottleneck, and (3) a 1.7B-parameter executor trained via quality-filtered trajectory distillation matches a frontier sub-agent in accuracy while consuming 37% fewer tokens. The experimental design is well-controlled: the answerer is held fixed, the retrieval corpus and retriever are constant, and the exclusion of instances answerable without search is a reasonable confound control.","tokens_in":16813,"tokens_out":1144,"duration_ms":236787,"significance":"The paper addresses a practically important question for multi-agent search system design—where to concentrate model capacity in a delegation–execution hierarchy. The role-factorized experimental instrument is clean, and the finding that execution can be aggressively downsized while delegation remains the bottleneck is actionable. The SFT pipeline for the compact executor is well-motivated by the behavioral gap analysis (Table 4: 1.7B does single-search 99.83% of the time vs. 40.31% multi-search for the teacher), and the quality-filtering design (preserving single-search competence, injecting multi-search correction with a counterfactual filter) is a concrete, falsifiable contribution. The cross-backbone generalization check (Table 5, GLM-5.1) and the Pareto frontier analysis (Figure 3) add credibility. Code is publicly available.","major_comments":[{"comment":"§3.3, Interventions 2 and 3; §4.5, Fig. 2: The central claim—that decomposition is the capability bottleneck—rests on comparing Δ_D ≈ 11 EM (Table 2, backbone varied, executor fixed at Qwen3-1.7B) against Δ_E ≈ 2.6 EM (Table 3, executor varied, backbone fixed at DeepSeek-V4-Flash). These two sweeps are measured at different operating points of the held-fixed variable: the delegation sweep holds the executor at its weakest (1.7B), while the execution sweep holds the backbone at its strongest (DeepSeek-V4-Flash). This design choice maximizes Δ_D while minimizing Δ_E, which is a classic interaction-effect problem. The two slopes are measured on different cross-sections of the (c_D, c_E) surface, and the asymmetry could be an artifact of where each cross-section is taken rather than a property of the surface itself. The paper does not acknowledge this confound or test its robustness. At a 2×","section":null}],"minor_comments":[{"comment":"§4.1: The LLM-as-judge uses DeepSeek-V4-Flash, which is also used as the frontier backbone and sub-agent in several experiments. This creates a potential self-preference bias in the judge scores. Consider noting this as a limitation.","section":null},{"comment":"Table 1: The EM gains from single-agent to main–sub are described as '4.5 to 8.6 points' in the abstract, but the table shows gains ranging from +4.52 to +8.63. Minor rounding inconsistency.","section":null},{"comment":"Figure 9 caption: The caption reads 'Delegation prompt (part 3): worked example of a multi-hop question with sequential sub-queries,' but the figure is labeled as the execution prompt (search sub-agent) in §C.2. Caption appears mismatched.","section":null},{"comment":"§4.4: The text states 'Scaling from Qwen3-1.7B to Qwen3-32B moves Avg. EM by only 1.27 points (from 39.18 to 40.45),' but Table 3 shows Qwen3-32B at 40.45 EM and Qwen3-1.7B at 39.18 EM, so the difference is 1.27. However, the abstract and §4.5 cite Δ_E ≈ 2.6, which appears to include the DeepSeek-V4-Flash sub-agent (41.77 EM). Clarify which comparison defines Δ_E.","section":null},{"comment":"App. D: The SFT corpus contains 2,168 records from 1,591 unique source questions. Consider reporting the single-search vs. multi-search split in the filtered corpus to verify that the multi-search correction behavior was adequately represented.","section":null}],"recommendation":"major_revision","confidential_remarks":"The stress-test concern about asymmetric operating points is the key issue for this paper. The central claim is defensible—the delegation axis likely does matter more than execution—but the current experimental design does not cleanly establish the magnitude of the asymmetry. A single additional experiment (execution sweep with a weak backbone, or delegation sweep with a strong executor) would either confirm or qualify the claim. I lean toward major revision rather than reject because the role-factorized framework, the SFT pipeline, and the Pareto analysis are valuable independent of whether the asymmetry is exactly 4× or somewhat smaller. The authors should be given the opportunity to address the interaction-effect confound, as it is fixable within the manuscript's scope."},"author_rebuttal":{"model":"glm-5.2","summary":"We thank the referee for a careful and constructive reading of our manuscript. The referee correctly identifies a genuine confound in our experimental design: the delegation and execution sweeps are measured at different operating points of the held-fixed variable, which could inflate the apparent asymmetry. We agree this is a real concern and will address it through additional experiments and revised framing in the revision.","responses":[{"response":"The referee raises a valid and important concern. We acknowledge that the two capacity sweeps are measured at different operating points of the held-fixed variable, and that this design could in principle amplify the apparent asymmetry between delegation and execution sensitivity. This is a genuine confound in our current experimental design, and we appreciate the referee identifying it so precisely. In the revision, we will address it in two ways. First, we will add additional cross-sections of the (c_D, c_E) surface: specifically, we will run the delegation sweep with the executor fixed at a stronger operating point (e.g., Qwen3-32B or DeepSeek-V4-Flash) and the execution sweep with the backbone fixed at a weaker operating point (e.g., Qwen3-8B or Qwen3-14B). This will test whether the asymmetry is robust to the choice of cross-section or whether it diminishes when the held-fixed variable is set at a more comparable level. Second, we will revise the manuscript to explicitly acknowledge the interaction-effect concern in §3.3 and §4.5, and we will qualify the central claim accordingly: rather than stating that decomposition is the bottleneck as an absolute property, we will frame it as holding across the operating points we test, pending the additional cross-sections. We expect the asymmetry to persist—because the case study in §4.7 provides a mechanistic explanation for why decomposition quality governs the performance ceiling regardless of executor strength—but we agree this must be demonstrated empirically rather than assumed. If the additional experiments show that the asymmetry narrows substantially at different operating points, we will revise the strength of the claim accordingly.","revision_made":"yes","referee_comment":"§3.3, Interventions 2 and 3; §4.5, Fig. 2: The central claim—that decomposition is the capability bottleneck—rests on comparing Δ_D ≈ 11 EM (Table 2, backbone varied, executor fixed at Qwen3-1.7B) against Δ_E ≈ 2.6 EM (Table 3, executor varied, backbone fixed at DeepSeek-V4-Flash). These two sweeps are measured at different operating points of the held-fixed variable: the delegation sweep holds the executor at its weakest (1.7B), while the execution sweep holds the backbone at its strongest (DeepSeek-V4-Flash). This design choice maximizes Δ_D while minimizing Δ_E, which is a classic interaction-effect problem. The two slopes are measured on different cross-sections of the (c_D, c_E) surface, and the asymmetry could be an artifact of where each cross-section is taken rather than a property of the surface itself. The paper does not acknowledge this confound or test its robustness."}],"tokens_in":16381,"tokens_out":658,"duration_ms":152921,"standing_objections":[]},"desk_editor":{"model":"glm-5.2","letter":"The main thing to know: this paper runs a clean controlled experiment showing that in hierarchical (main-agent → sub-agent) search, scaling the delegation backbone matters far more than scaling the executor. That finding is new and the experimental discipline is better than most work in this space — fixed answerer, fixed retriever, fixed corpus, exclusion of no-search-needed instances, no-retrieval control to check for contamination. Code is released. The SFT pipeline for the compact executor is well-motivated: the behavioral analysis (Table 4) showing the 1.7B model does single-search 99.83% of the time vs. 40.31% multi-search for the teacher is a concrete, useful diagnostic, and the quality-filtering strategy (preserve single-search competence, inject multi-search correction with a counterfactual filter) is sensible. The generalization check across two backbones (Table 5) is a nice touch. The no-retrieval control (App. A) showing DeepSeek at 36.65 EM without retrieval is a legitimate contamination concern, and the authors' choice to use Qwen3-32B as the fixed answerer instead is well-justified. Now the soft spots. The stress-test concern about asymmetric operating points is partially valid but overstated. The delegation sweep fixes the executor at its weakest (1.7B) while the execution sweep fixes the backbone at its strongest (DeepSeek-V4-Flash). This does mechanically inflate Δ_D and deflate Δ_E. However, the paper's own Table 5 provides a partial counter: the SFT executor under the GLM-5.1 backbone shows the same pattern, and the execution sweep's near-flatness across five model scales (1.7B through 32B plus frontier) is hard to explain purely as an operating-point artifact — if execution capacity mattered more at weaker backbones, we'd expect some slope. The real gap is that they never populate the off-diagonal cells (weak backbone + strong executor; strong backbone + weak executor is covered). One diagonal would settle this. Without it, the 4× asymmetry ratio is an upper bound, not a point estimate. The claim that SFT-Qwen3-1.7B 'matches' the frontier sub-agent is borderline: the margin is 0.04 EM and 0.48 judge with mean@4 sampling and no significance test. It's probably real but not demonstrated rigorously. SFT hyperparameters are missing from the paper. The study is limited to English multi-hop QA over Wikipedia — the authors acknowledge this. This paper is for practitioners building multi-agent search systems and researchers studying role decomposition in agents. The central finding (delegation is the bottleneck, execution can be compressed) is likely robust even if the exact ratio is inflated. It deserves a serious referee who should push for: (1) at least one off-diagonal cell, (2) significance tests or more rollouts for the SFT-vs-frontier comparison, (3) SFT training details. The core contribution survives these fixes.","headline":"Controlled capacity sweep shows delegation matters more than execution in hierarchical search agents; the asymmetry ratio is real but the sweep design inflates it somewhat.","tokens_in":17690,"tokens_out":697,"would_cite":true,"duration_ms":149675,"reading_group":"no","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"glm-5.2","headline":"Delegation Drives Multi-Agent Search, Not Execution","keywords":["multi-agent search","capacity allocation","hierarchical agents","task decomposition","trajectory distillation","multi-hop question answering","Pareto frontier","role factorization"],"falsifier":"If a weaker answerer or one with access to raw passages were substituted, execution capacity might become the dominant factor in end-task accuracy.","tokens_in":16631,"feed_emoji":"🔍","tokens_out":1095,"duration_ms":140855,"temperature":0.7,"pith_summary":"This paper studies how to allocate model capacity across roles in a hierarchical search agent—a system where a main agent decomposes a complex question into sub-queries and dispatches them to parallel sub-agents for retrieval. The authors factorize the task into three roles: delegation (task decomposition), execution (retrieval and evidence extraction), and answer generation (held fixed as a control). Through controlled capacity sweeps on five multi-hop QA benchmarks, they find a sharp asymmetry: scaling the delegation backbone improves exact match by about 11 points, while scaling the execution sub-agent moves it by only about 2.6 points. This identifies decomposition—not retrieval quality—as the capability bottleneck. Building on this, the authors train a 1.7B-parameter executor via quality-filtered trajectory distillation that matches a frontier-scale sub-agent in accuracy while consuming 37% fewer tokens, demonstrating that execution can be radically downsized without sacrificing performance. The paper's central claim is that the capability boundary of a hierarchical search agent is governed not by uniform component strength but by where capacity is placed in the hierarchy.","feed_headline":"In Multi-Agent Search, Delegation Matters 4x More Than Execution","feed_subtitle":"Scaling the planning backbone boosts accuracy 11 points; scaling the executor moves it 2.6. A 1.7B executor matches a frontier model at 37%","key_machinery":"The experimental design holds the answer-generation model fixed (Qwen3-32B, chosen for low parametric contamination) and varies only one of the two studied roles at a time. The delegation-capacity sweep fixes the executor at 1.7B and scales the backbone; the execution-capacity sweep fixes the backbone at a frontier model and scales the executor. The contrast between these two sweeps localizes the bottleneck. The compact executor training pipeline uses quality-filtered trajectory distillation from a stronger teacher, with a counterfactual filter that retains multi-search demonstrations only when a single-search variant of the same query fails—ensuring the student learns genuine iterative-retr","core_discovery":"The paper's central discovery is the capacity sensitivity asymmetry between delegation and execution in hierarchical search agents. Scaling the backbone (the model that decomposes questions and plans sub-queries) yields roughly four times the accuracy gain of scaling the executor (the model that retrieves and reads documents), across the same range of model scales. This asymmetry means the Pareto frontier of accuracy-versus-cost favors concentrating capacity in the backbone while keeping the executor compact. A 1.7B-parameter executor trained with targeted trajectory distillation—preserving single-search competence while injecting multi-search correction behavior—matches a frontier sub-agent","pith_inferences":["The bottleneck attribution depends on the answerer being strong enough to synthesize correct answers from sub-query reports alone. If the answerer were weaker or had access to noisier evidence, execution quality might become a larger bottleneck, since the system would rely more on the executor to pre-filter and organize evidence.","The asymmetry might shift on open-web search tasks (as opposed to the fixed Wikipedia corpus used here): if retrieval is noisier and documents less structured, execution quality—particularly multi-search correction and evidence extraction—could become more capacity-sensitive.","The same factorization logic may apply to other hierarchical agent systems beyond search (e.g., coding agents, tool-use agents): if decomposition is the bottleneck there too, compact specialized executors trained via in-framework distillation could replace frontier models in those executors.","The counterfactual filtering technique—retaining multi-step demonstrations only when the single-step variant fails—could be a general principle for distilling iterative-correction behavior without teaching spurious retries."],"forward_implications":["Practitioners building multi-agent search systems should allocate their largest, most capable model to the delegation role and use a compact specialized executor, rather than instantiating all roles from one shared model.","A 1.7B executor trained with in-framework trajectory distillation can match a frontier sub-agent at a fraction of the token cost, making hierarchical search substantially cheaper to deploy.","The finding that off-the-shelf end-to-end search agents (like SearchR1) underperform when placed in the executor role suggests that execution-role training data must match the input distribution of atomic sub-queries, not full multi-hop questions.","The asymmetry implies that improving a search agent's capability is more efficiently achieved by improving decomposition quality—better relational reasoning, better sub-query scoping—than by improving retrieval or reading comprehension."],"fun_headline_variants":["In Multi-Agent Search, Delegation Capacity Matters 4x More Than Execution","Big Planner, Small Executor: How to Size Multi-Agent Search Roles","Scaling Delegation Beats Scaling Execution 4x in Hierarchical Search","1.7B Executor Matches Frontier Sub-Agent in Hierarchical Search","Delegation Capacity Is the Bottleneck in Multi-Agent Search"],"cache_read_input_tokens":0,"weakest_assumption_plain":"The claim that decomposition is the bottleneck depends on the experimental design holding the answer-generation model fixed and restricting its input to only the question and sub-query reports. If the answerer were weaker, or if it had access to raw retrieved passages or reasoning traces, execution quality might become a larger bottleneck because the answerer would need to do more synthesis work from noisier evidence.","fun_headline_variants_meta":{"raw":{"variants":["In Multi-Agent Search, Delegation Capacity Matters 4x More Than Execution","Big Planner, Small Executor: How to Size Multi-Agent Search Roles","Scaling Delegation Beats Scaling Execution 4x in Hierarchical Search","1.7B Executor Matches Frontier Sub-Agent in Hierarchical Search","Delegation Capacity Is the Bottleneck in Multi-Agent Search"]},"model":"glm-5.2","effort":"high","cost_usd":0.0,"raw_usage":{"total_tokens":692,"prompt_tokens":618,"completion_tokens":74,"prompt_tokens_details":null},"tokens_in":618,"tokens_out":74,"duration_ms":32766,"temperature":1.0,"reasoning_tokens":null,"cache_read_input_tokens":0,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-09T07:10:53.088083+00:00","model_set":{"reader":"glm-5.2"},"falsifier":"If a weaker answerer or one with access to raw passages were substituted, execution capacity might become the dominant factor in end-task accuracy.","supporting_citations":[],"review_version":1}