{"id":"33474618-4319-4bcc-a980-59e4bdb9b7ef","arxiv_id":"2607.25656","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"OrchBench evaluates multi-agent orchestration plans by deterministic simulation on task DAGs, claiming strong correlation with real execution (r=0.816) at a small fraction of the cost.","lead":"OrchBench is a simulator that scores multi-agent orchestration plans by replaying them on task dependency graphs without running worker agents. Its simulated quality scores correlate with real Claude Code executions at r=0.816 while using about 1.3% of the tokens, and the authors use it to argue that preserving information across agents matters more than adding agents.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Simulator fidelity rests on six model-level points; leave-one-out r drops to 0.421, and no plan-level validation is reported, so the r=0.816 claim is not robust.","rationale":"The paper is honest and transparent: it reports leave-one-out, failure cases, ablations, and even shows that the headline 'coverage matters' effect disappears when the penalty is removed. The core issue is not the authors' integrity but the strength of the evidence for the central claim. The r=0.816 is computed at the level of model means (Appendix D-I), not individual plans. MultiAgentBench has many tasks per model; averaging can suppress noise and make an unstable plan-level relationship look strong. The leave-one-out spread (0.421–0.949) shows the result hinges on a single model, which is exactly the type of fragility that should block a strong claim. A plan-level analysis is the obvious missing piece: if the simulator really measures orchestration quality, it should rank individual plans executed by the same worker (Claude Code), not merely rank model averages. The hand-set parameters are another aspect of the same weakness: they are plausible but not identified from data; a small grid over γ or the exponent in Eq. 3 might change the correlation substantially. We agree with the reader's conditional verdict: the benchmark idea is useful, but the central fidelity claim needs stronger, plan-level validation and robustness analysis before the benchmark can be trusted for screening. Therefore verdict should remain CONDITIONAL (i.e., no change from the reader).","tokens_in":27973,"tokens_out":8216,"duration_ms":91441,"concrete_test":"On MultiAgentBench, collect per-plan real execution quality and the corresponding OrchBench simulated final score for at least 50–100 (task, plan) pairs (the six models × tasks already run likely provide this), and compute Pearson/Spearman correlation at the plan level, together with a bootstrap/permutation confidence interval. In addition, run a bootstrap over the six model means to obtain a CI for the model-level r. If the plan-level correlation is not significantly positive, or if the model-level 95% CI extends below 0.5, the central fidelity claim is not established; a reliable simulator should survive both checks.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that OrchBench's simulated scores can stand in for real execution quality—is supported only by a model-level correlation of r=0.816 over six models (Table 2; p=0.047). Leave-one-out correlations range from 0.421 to 0.949 (Table 3): removing a single model (Doubao-Mini) drops r below significance. Because the benchmark's purpose is to compare individual orchestration plans, the relevant evidence is plan-level, but no plan-level correlation is reported. Averaging many runs per model can inflate correlation and can hide systematic failures of the simulator on particular plans. Moreover, the quality computation (Eq. 3, geometric mean of parent qualities clipped and exponentiated by e^{s_v}) and the parameters λ=0.5 and γ∈{0.35,0.85,1.25} are hand-set rather than estimated from real data; the reported grid only varies λ (Table 20), not γ or the functional form. Finally, the headline result that transfer coverage matters more than agent count is partly an artifact of the missing-transfer penalty (λ multiplies parent quality, lowering all downstream q_v), so this 'finding' is not independently validated by real execution. The load-bearing assumption—that Eq. 3 faithfully models information-loss propagation—therefore rests on a single, fragile aggregate correlation.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces OrchBench, a deterministic-simulation benchmark for evaluating multi-agent orchestration plans without invoking worker agents. Given a task DAG, a context limit, and an agent budget, a planner produces an assignment and a set of cross-agent information transfers; the simulator (Algorithm 1) propagates quality through Eq. (3), applies context compression, and returns quality, makespan, and token cost. The paper reports a model-level Pearson correlation of r=0.816 between simulated scores and Claude Code executions on MultiAgentBench, claims large token/time savings, and uses the simulator to argue that transfer coverage matters more than agent count and that parallelism yields diminishing returns as coordination failures accumulate.","tokens_in":28284,"tokens_out":5201,"duration_ms":55106,"significance":"The problem is timely and the proposed isolation of orchestration from worker execution is potentially valuable. The simulator is explicit and deterministic, the metric definitions are mostly clear, and the paper includes honest sensitivity analyses and failure-case discussions. If the simulator were validated at the level of individual plans, OrchBench could become a useful low-cost screening tool for orchestration research. However, the current validation evidence is too thin to establish the central claim that simulated scores can stand in for real execution quality.","major_comments":[{"comment":"The central validation is a Pearson r=0.816 over six model-level averages (Table 2; p=0.047). Table 3 shows leave-one-out r ranging from 0.421 (removing Doubao-Mini) to 0.949; removing that single model makes the correlation non-significant (p=0.500). Because OrchBench claims to evaluate individual orchestration plans, the relevant validity evidence is plan-level, but no plan-level correlation is reported anywhere: Table 1 and Table 2 both aggregate runs per model. Model-level averaging can inflate a correlation by suppressing within-model plan variance and can hide systematic failures on specific plans. Please provide per-plan/per-task simulation-to-real correlations, with many more than six comparison points, and report scatter and pairwise plan-ranking accuracy.","section":"Benchmark Validation (Tables 2-3)"},{"comment":"The quality model in Eq. (3) sets q_v = clip(geometric mean of available parent qualities)^{e^{s_v}}, and Algorithm 1 (lines 15-17) applies lambda=0.5 to every missing transfer. Consequently, any plan with more missing transfers has lower Q by construction. This makes the headline result in Table 5 ('coverage matters more than agent count') partly a mechanical consequence of the simulator, not an independently validated empirical finding. To support the claim, the paper needs (i) real-execution evidence that missing transfers and compression losses propagate approximately multiplicatively, and (ii) a sensitivity analysis over the functional form and over gamma, not only over lambda (Table 20). The compression-sensitivity classes (robust/balanced/fragile) are also hand-defined; their mapping to e^{gamma} with gamma in {0.35,0.85,1.25} should be justified or estimated.","section":"Deterministic Simulation, Eq. (3); Algorithm 1 (lines 15-17)"},{"comment":"The r=0.816 correlation is for Score = (Q + E_time + E_token)/3 (Eq. 7). Table 2 reports r=-0.264 for time and r=-0.607 for token usage, and Appendix D-II states that time and token consumption are framework-dependent and cannot be reliably predicted. Since the aggregate score contains two components that are negatively or weakly correlated with real outcomes, the positive final-score correlation may be driven almost entirely by Q. Please report the correlation of each component separately and explicitly state whether E_time and E_token are intended as predictive or only as diagnostic. As written, the final score can mislead users into thinking the simulator predicts all three dimensions.","section":"Outcome alignment (Table 2); Eq. (6)-(7)"},{"comment":"Table 9 reports that simulator-guided refinement raises mean real-execution score from 3.754 to 4.150 on 20 tasks. No standard errors, confidence intervals, or paired significance tests are given, and the refinement inserts only one handoff selected from simulation. Because this table is used to support the claim that simulator-guided diagnosis improves real execution, it needs a proper paired test with per-task breakdown; the current evidence is anecdotal.","section":"Simulator-guided refinement (Table 9)"}],"minor_comments":[{"comment":"The exact correlation coefficients are embedded in a heatmap with upper and lower triangles. Please provide a table with the exact values and explain in the caption that the upper/lower triangles report Pearson/Spearman coefficients.","section":"Figure 3"},{"comment":"With six models, many p-values are not corrected for multiple comparisons. Treat the structural-alignment correlations as descriptive and state this explicitly.","section":"Table 1"},{"comment":"The mapping from token budgets to the context limit L is described only in the appendix. A one-sentence summary in the main text would help readers understand the context-pressure mechanism.","section":"Section G (Other Settings)"},{"comment":"The compression-sensitivity class (robust/balanced/fragile) is introduced in the problem formulation but is not operationalized until Appendix G. Define it near Eq. (2) with a small example.","section":"Eq. (2)-(3)"},{"comment":"'ZeroScore' is returned for invalid plans but is not defined. Clarify whether all metrics are zero or only the final score.","section":"Algorithm 1"}],"recommendation":"major_revision","confidential_remarks":"The proposed benchmark is within the journal's scope and the simulator design is transparent. The main obstacle is validation: the headline r=0.816 rests on six model-level points and is not robust to leave-one-out. I would require plan-level validation and a direct test of the quality-propagation model before acceptance. No concerns about novelty or citation behavior."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a genuinely useful benchmark idea, presented with unusual transparency, but the headline validation is thinner than it looks and one of the headline \"findings\" is partly built into the simulator. Worth sending to referees, with the expectation of major revision on validation and framing.\n\nWhat's new: OrchBench is the first benchmark I know that isolates orchestration-plan quality from worker execution using a deterministic simulator. That is a real gap, and the design is thoughtful: DAGs constructed from real tasks with controlled parallelism, planner-produced transfer and compression decisions, and a simulator that returns quality, makespan, and token costs without invoking workers. The simulator itself is explicit and deterministic (Algorithm 1), and the paper is admirably honest about sensitivity analyses, leave-one-out correlations, DAG generation failures, and cost-accounting choices. The simulator-guided refinement experiment, where a simulator-selected handoff improves real execution scores, is a genuinely useful external check.\n\nSoft spots: the central fidelity claim rests on six model-level points (r=0.816, p=0.047), and leave-one-model-out drops to 0.421 when Doubao-Mini is removed. No plan-level sim-to-real correlation is reported, even though the benchmark's purpose is comparing plans, not just ranking models. The parameters lambda and gamma are hand-set; the sensitivity sweep varies lambda only. The \"coverage matters more than agent count\" result is partly encoded in Eq. 3 and Algorithm 1, because missing transfers mechanically multiply down the quality of all downstream tasks. The ablation confirms this: without the missing-transfer penalty, model separation collapses. That does not invalidate the benchmark — it shows the mechanism is doing what it was designed to do — but the paper should not present the result as an empirical discovery. It is a design property with only weak external validation. Also, no code or data are released, which makes reproduction harder.\n\nWho this is for: people building or evaluating multi-agent orchestration frameworks. The core idea is solid and the execution is honest, so a serious referee should engage with it. The revision needs plan-level validation, more models, artifact release, and sharper framing of what is discovered versus what is constructed.","headline":"A genuinely useful simulation-based orchestration benchmark, honestly reported, but the headline validation is thin (six models, leave-one-out r drops to 0.421) and the 'coverage beats agent count' finding is partly built into the simulator rather than independently discovered.","tokens_in":28812,"tokens_out":2045,"would_cite":true,"duration_ms":26190,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that orchestration plans for multi-agent systems can be scored by deterministic simulation alone, with simulated scores tracking real execution quality at r=0.816 while using about 1.3% of the tokens and 10.3% of the time.","keywords":["multi-agent orchestration","deterministic simulation","benchmark","DAG workflows","information transfer","context compression","orchestration evaluation","coordination failures"],"falsifier":"Gather per-subtask output-quality ratings from real agent executions on a sample of OrchBench DAGs and compare them node-by-node with the simulator's q_v values. If node-level agreement is weak despite the aggregate r=0.816, the geometric-mean propagation rule is not the actual mechanism and the benchmark's rankings could fail on out-of-distribution plans.","tokens_in":27823,"feed_emoji":"🧩","tokens_out":7049,"duration_ms":70694,"temperature":0.7,"pith_summary":"OrchBench aims to let researchers judge the orchestration layer of multi-agent systems—who does which subtask, in what order, and what information gets handed off—without paying the cost of actually executing the agents. The paper's central claim is that a deterministic simulator, fed only the task dependency graph and the orchestration plan, can reproduce the ordering of real execution outcomes: simulated quality scores correlate with real workflow quality at Pearson r=0.816 while using roughly 1.3% of the tokens and 10.3% of the wall-clock time. If that holds, orchestration quality becomes a fast, reproducible, and interpretable measurement rather than a byproduct of end-to-end runs. Using the simulator at scale, the paper finds that successful information transfer—not the number of agents—is what keeps multi-agent quality high, and that extra agents yield diminishing returns once coordination failures and context pressure set in. A sympathetic reader would take away that plan-level evaluation is feasible today, and that the bottleneck in current multi-agent orchestration is information preservation across handoffs.","feed_headline":"Simulation alone scores agent plans, matching real runs at r=0.816","feed_subtitle":"OrchBench's deterministic simulator needs only 1.3% of tokens and 10.3% of the time, and exposes missing handoffs.","key_machinery":"The load-bearing object is the deterministic simulator, in particular its quality-propagation rule: each subtask's output quality is the geometric mean of the retained qualities of its parent results, raised to a compression-sensitivity exponent and clipped to [qmin,1]; a missing cross-agent transfer multiplies the parent's contribution by a fixed penalty factor. Context compression reduces token count and quality according to retention ratio and the subtask's fragility class. This rule is what lets the benchmark translate a plan into a quality number without invoking workers, and it is also the part of the pipeline that must be faithful for the benchmark's rankings to transfer to real execu","core_discovery":"The paper constructs a benchmark in which real-world tasks are converted into directed acyclic graphs of subtasks with token, time, and compression-sensitivity metadata. A planner must assign each subtask to an agent and declare cross-agent information transfers with retention ratios; a deterministic simulator then replays the workflow—dependency resolution, agent scheduling, context acquisition and compression, execution, state updates—and outputs final quality, makespan, and token cost. The central claim is that this simulated evaluation captures the orchestration-relevant part of real execution: across real tool-based runs, simulated scores and real task quality correlate with Pearson r=0","pith_inferences":["A natural next step beyond this paper would be to search over orchestration plans against the deterministic simulator before executing anything, turning orchestration into an optimizable planning problem rather than a one-shot model judgement.","The 'coverage matters more than agent count' result is correlational within the simulator; a stronger test would hold transfer coverage fixed and vary agent count in real executions to confirm the causal direction it implies.","The quality-propagation rule is a strong hypothesis about information loss; collecting per-subtask real output-quality ratings would let future work test whether the geometric mean is the right composition law or whether a minimum or additive rule fits real degradation better.","Because the final score averages quality, speed efficiency, and token efficiency with equal weights, the reported model rankings are sensitive to those weights; a weight-sensitivity analysis would show how much of the headline ordering reflects the benchmark's objective rather than orchestration skill."],"forward_implications":["Orchestration plan quality can be measured in isolation, so researchers can compare planners under identical conditions without worker, tool, or environment noise.","Simulated scores predict real execution quality well enough for early screening: r=0.816 overall, positive in all leave-one-model-out checks (r=0.421–0.949).","Evaluation cost drops by orders of magnitude—1.3% of tokens and 10.3% of wall-clock time on the validation set, with up to 389× token and 110× time reductions on individual tasks.","At scale, missing information transfers, not agent count, dominate quality degradation; the benefit of additional agents saturates and can turn negative once the working state fits a single context window.","Simulator-guided handoff repair improves real workflow scores (3.754 to 4.150 out of 5 on the refinement test), supporting use of the simulator as a diagnostic tool."],"fun_headline_variants":["Agent plans scored without execution at 1.3% tokens","Orchestration gaps exposed by deterministic simulation","Simulation predicts quality at r=0.816 using 1.3% tokens","Preserving critical info beats adding agents in orchestration","Benchmark evaluates orchestration plans without running agents"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The simulation's quality equation—geometric mean of parent qualities, with missing transfers discounted by a fixed factor and hand-set compression exponents—is assumed to mirror how real information loss compounds into final task quality; the paper validates this only at the level of whole-model aggregates on six models, not at the level of individual subtasks.","fun_headline_variants_meta":{"raw":{"variants":["Agent plans scored without execution at 1.3% tokens","Orchestration gaps exposed by deterministic simulation","Simulation predicts quality at r=0.816 using 1.3% tokens","Preserving critical info beats adding agents in orchestration","Benchmark evaluates orchestration plans without running agents"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000383,"raw_usage":{"total_tokens":1892,"prompt_tokens":797,"completion_tokens":1095,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":541,"completion_tokens_details":{"reasoning_tokens":1013}},"tokens_in":541,"tokens_out":1095,"duration_ms":10057,"temperature":1.0,"reasoning_tokens":1013,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-01T01:48:32.170022+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Gather per-subtask output-quality ratings from real agent executions on a sample of OrchBench DAGs and compare them node-by-node with the simulator's q_v values. If node-level agreement is weak despite the aggregate r=0.816, the geometric-mean propagation rule is not the actual mechanism and the benchmark's rankings could fail on out-of-distribution plans.","supporting_citations":[],"review_version":1}