Pith. sign in

REVIEW 4 major objections 6 minor 18 references

Under equalized prompt optimization, multi-call LLM orchestration buys modest, benchmark-dependent accuracy gains while consuming roughly 2–4 times the tokens of a single call.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Across five LLMs and three benchmarks, orchestration adds up to 4.6 points over optimized single-call CoT at 2-4x token cost, with no difficulty-scaled benefit but strong method-by-backbone interactions.

T0 review reviewed 2026-08-05 challenge →

load-bearing objection Careful orchestration-cost study, but the 'controlled' claim is weakened by optimizing prompts only on DeepSeek and transferring them. the 4 major comments →

arxiv 2608.00685 v1 pith:GOCSOIDI submitted 2026-08-01 cs.AI

When Does LLM Orchestration Pay Off? A Controlled Evaluation of Accuracy, Cost, and Task Difficulty

Nicolas Leins , Nico Pelleriti , Jana Gonnermann-M\"uller , Sebastian Pokutta This is my paper
classification cs.AI
keywords LLM orchestrationinference-time computeprompt optimizationtask difficultyaccuracy–cost trade-offSelf-RefineBest-of-Nmulti-agent debate
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether LLM orchestration—running several model calls that refine, select, or debate each other—earns its extra inference cost. Across five open-weight LLMs and three problem domains, with every method's prompts optimized under the same GEPA budget, the authors find that Self-Refine and Best-of-N beat optimized chain-of-thought on competitive programming and math by up to 4.6 percentage points, but that no orchestration helps on chess puzzles and Debate never beats CoT. The gains come at roughly 2–4 times the mean token use of a single-call baseline. Human-derived item difficulty predicts lower absolute accuracy but not larger orchestration benefits; instead, the value of orchestration depends heavily on the backbone model. The paper concludes that orchestration should be chosen per model and judged by accuracy–cost trade-offs, not treated as uniformly beneficial.

Core claim

The paper's central claim is that, once prompt-optimization effort is controlled, LLM orchestration offers moderate and benchmark-dependent accuracy gains that do not scale with task difficulty and vary strongly with the base model. Averaged over five backbones, the largest improvements are +4.6 points over optimized CoT (Self-Refine on Codeforces) and +4.5 over task-only inference, while some methods hurt specific models. The authors interpret this as evidence that additional inference-time structure is not intrinsically valuable: revision, selection, and debate help only when the backbone can supply the needed capability (self-diagnosis, candidate diversity, critique synthesis), so orchest

What carries the argument

GEPA, an automated prompt-evolution procedure, is used to optimize every method's prompt components under a common budget of 750 metric calls and a per-benchmark weighted-token cap; this is the mechanism that makes the comparison controlled. On top of it, the evaluation uses difficulty-stratified fixed item subsets, paired whole-item bootstrap inference, and mixed logistic models with random item and item-by-LLM intercepts; likelihood-ratio tests distinguish average method effects (M1), method-by-difficulty interactions (M2), and method-by-LLM interactions (M3).

Load-bearing premise

The paper's comparison rests on the assumption, flagged in its Limitations section, that GEPA under the same maximum budget optimizes all workflows equally well; if instead GEPA tunes some methods (or the hand-crafted AMC exception) better than others, measured orchestration gains could partly reflect prompt quality rather than orchestration structure.

What would settle it

A rerun in which each method's prompts are optimized to validation-accuracy saturation—or to the same number of candidate evaluations per tunable component—would settle the attribution; if orchestration-minus-CoT differences then collapse, the common-budget protocol (not orchestration structure) caused the gains.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • A validated single-call CoT baseline is the right efficiency default: orchestration should be adopted only when its moderate accuracy gain justifies the 2–4x token, latency, and serving cost.
  • Difficulty-aware routing systems cannot assume hard items automatically deserve more orchestrated calls; difficulty predicts which items are hard, not which workflow will fix them.
  • Orchestration evaluation should be done per model–workflow pair and report model-specific accuracy–cost trade-offs; a workflow can be strongly beneficial for one backbone and harmful for another.
  • Prompt optimization is part of the method, not a background detail: orchestration comparisons should fix an optimization budget, or reported gains conflate structure and tuning effort.
  • Debate, in this controlled setting, does not significantly beat CoT on any benchmark, so richer interaction alone does not guarantee improvement.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • A testable extension is to calibrate optimization effort per method by saturating validation accuracy (e.g., run GEPA until no candidate improves for several generations) and see whether the remaining orchestration gaps shrink or disappear; the paper's own validation-gain table (e.g., +62 points for CoT versus +0 for Debate on Codeforces) suggests at least part of the measured contrast may be prom
  • The method-by-backbone interaction pattern suggests orchestration value tracks component capabilities (self-correction, sampling diversity, critique) that could be measured directly; an even more controlled design would ablate orchestration components against a single-call baseline within each backbone.
  • The finding implies that automated workflow search systems should include a single-call optimized baseline in their search space and use model-specific cost-aware objectives, otherwise they may select multi-agent structures that only pay off on particular backbones.
  • Because human difficulty did not moderate gains, the paper implicitly narrows the justification for difficulty-based compute allocation: difficulty signals may select model capacity or workflow structure jointly, but not dictate more calls for fixed workflows.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper reports a controlled empirical evaluation of three LLM orchestrations (Self-Refine, Best-of-N, Debate) against task-only and chain-of-thought baselines, across five open-weight LLMs and three benchmarks (Codeforces, Lichess, AMC). Prompts are optimized with GEPA under common metric-call and weighted-token budgets, and all methods are evaluated on the same difficulty-stratified 200-item subsets. The authors find moderate, benchmark-dependent accuracy gains for Self-Refine and BoN over optimized CoT on Codeforces and AMC (up to +4.6 percentage points), at roughly 2–4 times the token cost of task-only inference, no significant orchestration benefit on Lichess, no evidence that orchestration effects increase with human-derived item difficulty, and strong method-by-backbone interactions.

Significance. The study has several genuine strengths: fully paired item-level data, prespecified confirmatory models M0–M2, 9,999 whole-item bootstrap replicates, Holm correction across planned contrasts, and complete model–method tables in the Supplement. The explicit measurement of token costs and the transparent documentation of the GEPA optimization protocol are also valuable. If the results hold, they provide a useful caution against treating orchestration structure as uniformly beneficial and support model-specific accuracy–cost evaluation. However, the central 'controlled optimization' claim is weakened by two protocol features: GEPA optimization is performed on a single backbone and then transferred to the other four, and the AMC benchmark uses a hand-crafted prompt exception. These issues directly affect the interpretation of the average gains and the method-by-LLM interactions. The difficulty conclusion is also tested only for linear trends. The paper is defensible but requires revision to address these load-bearing threats.

major comments (4)
  1. [§3.4, §3.2, §4.1] GEPA optimization is performed using DeepSeek V4 Flash as task, profile, and reflection model, yet the selected prompts are then applied to all five backbones. The optimization budget therefore ensures comparable prompt quality only for DeepSeek; for Gemma, GLM, Qwen, and GPT-OSS the prompts are a zero-cost transfer. The magnitude of possible prompt effects is large: Table 2 shows Codeforces CoT validation accuracy moving from .20 (seed) to .82 (selected), while Debate stays at .80, and §4.1 reports transferred CoT-minus-task-only differences from −10.5 to +10.0 points on Codeforces. These magnitudes mean the reported average orchestration-vs-CoT effects and the M3 method×LLM interactions (§4.4) could partly reflect differential prompt transfer across methods rather than orchestration structure. The Limitations section (§6) acknowledges unequal optimization across components but does not
  2. [§3.4, §4.2, Table 6] For AMC, the evaluated prompts are not produced under the common GEPA budget: the text states that 'GEPA did not improve on the near-saturated seed configuration, so we instead use a hand-crafted prompt.' AMC is one of the two benchmarks where significant orchestration gains over CoT are reported (Self-Refine OR 1.99, BoN OR 2.68; Table 6). Because the hand-crafted prompts are method-specific and manually chosen, the AMC evidence for the central claim is outside the controlled optimization protocol. The paper discloses this exception but does not bound its impact; the conclusion that Self-Refine and BoN outperform optimized CoT on 'Codeforces and AMC' should therefore be qualified or supported by additional analysis.
  3. [§3.4] The stated 'same maximum optimization budget' is defined by total metric calls and weighted-token caps, but the number of tunable prompt components differs across methods: CoT has one component, while Self-Refine, BoN, and Debate each have three (Section 3.3, Table 1). With 750 metric calls, CoT receives hundreds of optimization iterations on a single component, whereas the orchestrations split the same total calls among three components. This can produce large, method-specific differences in achievable prompt quality, as illustrated by the Codeforces GEPA outcomes in Table 2. Equalizing total budget is a defensible design choice, but it does not equalize per-component optimization effort; a sensitivity analysis with per-component budgets or a larger total budget would help establish whether the orchestration-vs-CoT comparisons reflect method structure or optimization headroom. The Limit
  4. [§4.3, Supplementary Fig. 7] The difficulty analysis tests whether orchestration effects grow with difficulty using only a linear method×difficulty interaction (M1 vs M2) and slope ratios from M2. The descriptive ten-bin trajectories are explicitly non-monotonic (e.g., AMC bin 9 increases to 98.0% after an earlier decline; Codeforces Q3 shows the largest orchestration-minus-baseline gains). A linear-only test can miss non-monotonic interactions, so the evidence does not rule out orchestration benefits that peak at intermediate difficulty. The main-text claim that 'the relative benefit of orchestration did not increase with item difficulty' is carefully worded, but the analysis is weaker than the headline suggests. The paper should either add non-linear or bin-based interaction tests or state the linear-only scope in the results section rather than only in the Limitations.
minor comments (6)
  1. [Abstract, §4.2] The abstract's '4.5 points over task-only' contrasts with an unoptimized task-only baseline, while CoT and the orchestrations are GEPA-optimized. Please clarify in the abstract or main text that the task-only comparison is descriptive and does not involve the same prompt-optimization protocol.
  2. [§3.4] The weighted-token ratio w=4 is an arbitrary cost assumption. A brief sensitivity analysis for w (e.g., w=1 and w=8) would make the token-efficiency ratios more robust; currently the choice is stated but not varied.
  3. [Figure 3] The cell labels in Figure 3 are dense and small. Consider moving the exact numbers to a table or using a larger font, and state in the caption that each cell is based on 200 paired items.
  4. [Table 1] The final column heading 'calls LLM' appears truncated or ambiguous; it should be something like 'Calls to backbone LLM per evaluation'.
  5. [§3.5] The handling of empty intermediate responses is described for BoN and Debate, but it is not stated how parse failures (e.g., no recognized selector statement in BoN, or a judge response that is not a valid final answer) are counted. Please clarify.
  6. [General] The paper does not mention a code/data release. Given the emphasis on reproducibility and the detailed prompts, a statement about the availability of evaluation code and generated prompts would strengthen the contribution.

Circularity Check

0 steps flagged

No circular derivation: the evaluation is empirical with disjoint validation/test splits; self-citations are contextual and not load-bearing.

full rationale

This paper does not present a formal derivation chain; it reports a controlled empirical evaluation. The core claims—accuracy differences between orchestration methods and single-call baselines, cost in weighted tokens, and difficulty interactions—are measured on held-out test items after GEPA prompt optimization on disjoint validation partitions (Sections 3.1, 3.4, 3.5). Test-set accuracy is not the optimization objective, so there is no fitted parameter renamed as a prediction. The mixed-model and bootstrap analyses (Section 3.7, Tables 5–7) are inferential summaries of held-out paired observations, not identities derived from the optimization inputs. Difficulty labels come from external human-derived ratings/IRT (Section 3.1), so the difficulty-related findings are not definitionally tied to the methods. The two self-citations (Zhou et al. 2026 and Zimmer et al. 2026) appear in the introduction as broad context about inference-time computation and LLM capabilities; neither is used to justify a central premise, to forbid alternatives, or to import a uniqueness theorem. The manuscript's stated limitations—single-backbone GEPA optimization transferred to four other LLMs, the AMC hand-crafted prompt exception, and unequal optimization of workflows with different component counts—are internal-validity concerns about whether optimization effort was truly equalized. They do not make the test-set results equivalent to the optimization inputs by construction. Therefore no circular step can be exhibited; the score reflects only the presence of minor, non-load-bearing self-citations.

Axiom & Free-Parameter Ledger

3 free parameters · 5 axioms · 0 invented entities

No new entities are postulated. The central results rest on design choices (GEPA budgets, w=4 weighting, orchestration call counts) and on the validation-driven prompt fits, which are the most consequential free parameters because they vary widely between methods.

free parameters (3)
  • weighted-token ratio w = 4
    Chosen by hand as an estimate of the average input-to-output token cost ratio (input + 4*(output+reasoning)); it defines the weighted-token cost metric used in Section 4.5, so reported cost ratios partially depend on this arbitrary choice.
  • GEPA optimization budgets = 750 metric calls; 40M (Codeforces) and 75M (Lichess) weighted tokens
    The common maximum budget is meant to equalize optimization effort across methods, but the number of tunable prompt components differs by method, so equal budgets may not imply equal optimization quality (see Limitations Section 6).
  • GEPA-selected method prompts = per-method optimized prompts on 50-item validation sets (Appendix B; seed and selected accuracies in Table 2)
    Each method's test accuracy is conditional on a prompt selected from 50-100 validation items. Seed-to-selected validation gains differ widely across methods (Codeforces CoT +62 pts, Debate 0 pts), so method effects are entangled with fitted prompt quality.
axioms (5)
  • domain assumption GEPA, under the fixed budgets, optimizes each orchestration's prompt components comparably, so accuracy differences reflect orchestration structure rather than unequal prompt tuning.
    Invoked in Section 3.4 as the basis for controlled comparison; contradicted in degree by Table 2 and acknowledged as uncertain in Limitations Section 6.
  • domain assumption Human-derived difficulty labels (Lichess and Codeforces ratings, AMC IRT estimates) are valid item-difficulty measures, comparable within each benchmark.
    Used for stratification and difficulty-slope analyses in Sections 3.1 and 3.7.
  • domain assumption Token consumption, weighted with w=4, is a reproducible proxy for inference cost.
    Section 3.6 defines tw and treats it as the resource metric; the cost interpretation of the accuracy-cost trade-off rests on this proxy.
  • standard math Mixed-effects logistic regression with random intercepts for item and item:LLM, and standard likelihood-ratio test approximations, gives valid inference for the planned contrasts.
    Section 3.7 and Appendix E; standard statistical practice, though LRT for fixed effects in GLMMs can be anti-conservative in small samples; n=4,000 observations here.
  • domain assumption The 200-item stratified subsets are representative of the E2H evaluation splits, and pass@1 grading (exact reference match or hidden tests) correctly scores correctness.
    Sections 3.1 and 3.5; subset sharing controls selection, but grading exactness (e.g., Lichess SAN/UCI exact match) is assumed by the benchmark.

reviewed 2026-08-05 · how reviews work

0 comments
Cite this review

Pith. "Pith review of When Does LLM Orchestration Pay Off? A Controlled Evaluation of Accuracy, Cost, and Task Difficulty." pith.science (2026). https://pith.science/paper/GOCSOIDI

@misc{pith2026260800685,
  author       = {Pith},
  title        = {Pith review of: When Does LLM Orchestration Pay Off? A Controlled Evaluation of Accuracy, Cost, and Task Difficulty},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/GOCSOIDI}},
  note         = {Machine review of arXiv:2608.00685}
}
Share X Bluesky LinkedIn Reddit HN
abstract

LLM orchestration is often assumed to improve reasoning by allocating additional inference-time computation, yet its gains may not justify its cost. Existing comparisons also frequently overlook differences in optimization effort, making it difficult to isolate the value of orchestration itself. We conduct a controlled evaluation of Self-Refine, Best-of-$N$, and Debate against task-only and chain-of-thought (CoT) single-call baselines across five LLM backbones and three domains: competitive programming, chess puzzles, and mathematics. For comparability, we optimize each method with GEPA under the same optimization budget and evaluate all methods on the same difficulty-stratified benchmark items. Orchestration yields moderate but benchmark-dependent gains: averaged across backbones within each benchmark, the largest improvement is 4.6 percentage points over optimized CoT inference and 4.5 points over task-only inference, while requiring approximately 2 to 4 times the mean total tokens of task-only inference. Human-derived difficulty is associated with lower absolute accuracy in all three benchmarks, but within-benchmark analyses do not indicate that orchestration effects increase with task difficulty. By contrast, exploratory mixed-effects analyses reveal strong interactions between orchestration method and backbone model across all three benchmarks, showing that orchestration effectiveness depends substantially on the underlying model. Our results suggest that orchestration decisions should be model-specific and account for whether moderate accuracy gains justify the additional inference cost. More broadly, evaluations of LLM orchestrations should control optimization effort and report model-specific accuracy--cost trade-offs rather than treating additional inference-time structure as uniformly beneficial.

Figures

Figures reproduced from arXiv: 2608.00685 by Jana Gonnermann-M\"uller, Nicolas Leins, Nico Pelleriti, Sebastian Pokutta.

Figure 1
Figure 1. Figure 1: Accuracy by method and benchmark, aggregated over the five evaluated models. Bars are point [PITH_FULL_IMAGE:figures/full_fig_p008_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Orchestration-minus-baseline accuracy by within-benchmark difficulty quartile. Differences are [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Model-specific orchestration-minus-CoT paired accuracy differences by benchmark. Each cell [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Accuracy against mean weighted tokens for each method and benchmark. Each point aggregates [PITH_FULL_IMAGE:figures/full_fig_p009_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Archived GEPA candidate trees for Codeforces. Node labels give candidate identifiers and validation [PITH_FULL_IMAGE:figures/full_fig_p014_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Archived GEPA candidate trees for Lichess. Node labels give candidate identifiers and validation [PITH_FULL_IMAGE:figures/full_fig_p014_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Absolute accuracy over ten within-benchmark bins of the continuous difficulty percentile. Each [PITH_FULL_IMAGE:figures/full_fig_p018_7.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

18 extracted references · 11 canonical work pages · 1 internal anchor

  1. [2]

    Towards reasoning era: a survey of long chain-of-thought for reasoning large language models.Science China Information Sciences, 69(6):161101, May 2026b

    Qiguang Chen, Libo Qin, Jinhao Liu, Dengyun Peng, Jiannan Guan, Peng Wang, Mengkang Hu, Yuhang Zhou, Te Gao, and Wanxiang Che. Towards reasoning era: a survey of long chain-of-thought for reasoning large language models.Science China Information Sciences, 69(6):161101, May 2026b. ISSN 1869-1919. doi: 10.1007/s11432-025-4665-8. URL https://doi.org/10.1007/...

    work page doi:10.1007/s11432-025-4665-8 1919

  2. [5]

    ISBN 979-8-4007-1926-4

    Association for Computing Machinery. ISBN 979-8-4007-1926-4. doi: 10.1145/3755881.3755925. URL https://dl.acm.org/ doi/10.1145/3755881.3755925. Mucong Ding, Chenghao Deng, Jocelyn Choo, Zichu Wu, Aakriti Agrawal, Avi Schwarzschild, Tianyi Zhou, Tom Goldstein, John Langford, Anima Anandkumar, and Furong Huang. Easy2Hard-Bench: Standardized Difficulty Label...

    work page arXiv 1926

  3. [6]

    pgn": SAN in standard algebraic notation **without any annotation symbols** (no ’+’ for check, no ’##’ for checkmate). Use uppercase letters for pieces. Example:

    **Output the move** as a JSON object with exactly: - "pgn": SAN in standard algebraic notation **without any annotation symbols** (no ’+’ for check, no ’##’ for checkmate). Use uppercase letters for pieces. Example: "Nxd6" not "Nxd6##". - "uci": UCI notation (e.g., "f5d6"). Do not include any other text or commentary. \par Ensure your reasoning is thoroug...

    work page arXiv

  4. [7]

    Zheng Du, Hao Kang, Song Han, Tushar Krishna, and Ligeng Zhu

    JMLR.org. Zheng Du, Hao Kang, Song Han, Tushar Krishna, and Ligeng Zhu. OckBench: Measuring the Efficiency of LLM Reasoning, February 2026b. URL http://arxiv.org/abs/2511.05722. arXiv:2511.05722 [cs.CL] version:

    Pith/arXiv arXiv

  5. [8]

    arXiv:2508.07407 [cs]

    URL http://arxiv.org/abs/2508.07407. arXiv:2508.07407 [cs]. Chengying Huan, Ziheng Meng, Yongchao Liu, Zhengyi Yang, Yun Zhu, Yue Yun, Shipeng Li, Rong Gu, Xiabao Wu, Haitao Zhang, Chuntao Hong, Shaonan Ma, Guihai Chen, and Chen Tian. Scaling Graph Chain-of-Thought Reasoning: A Multi-Agent Framework with Efficient LLM Serving, November

    Pith/arXiv arXiv

  6. [9]

    arXiv:2511.01633 [cs]

    URL http://arxiv.org/abs/2511.01633. arXiv:2511.01633 [cs]. Jie Huang, Xinyun Chen, Swaroop Mishra, Huaixiu Steven Zheng, Adams Wei Yu, Xinying Song, and Denny Zhou. Large Language Models Cannot Self-Correct Reasoning Yet. October

    arXiv

  7. [10]

    doi: 10.1038/s42256-026-01268-y

    ISSN 2522-5839. doi: 10.1038/s42256-026-01268-y. URL https://www.nature.com/articles/s42256-026-01268-y. Takeshi Kojima, Shixiang (Shane) Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large Language Models are Zero-Shot Reasoners. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors,Advances in Neural Information Processing ...

    work page doi:10.1038/s42256-026-01268-y

  8. [12]

    12 Isaac Ong, Amjad Almahairi, Vincent Wu, Wei-Lin Chiang, Tianhao Wu, Joseph E Gonzalez, M Kadous, and Ion Stoica

    URL https://openreview.net/forum?id=S37hOerQLB. 12 Isaac Ong, Amjad Almahairi, Vincent Wu, Wei-Lin Chiang, Tianhao Wu, Joseph E Gonzalez, M Kadous, and Ion Stoica. RouteLLM: Learning to Route LLMs from Preference Data. In Y. Yue, A. Garg, N. Peng, F. Sha, and R. Yu, editors,International Conference on Learning Representations, volume 2025, pages 34433–34448,

    work page 2025

  9. [13]

    Andries Smit, Nathan Grinsztajn, Paul Duckworth, Thomas D

    URL https://proceedings.iclr.cc/paper_files/paper/2025/file/5503a7c69d48a2f86fc00b3dc09de686- Paper-Conference.pdf. Andries Smit, Nathan Grinsztajn, Paul Duckworth, Thomas D. Barrett, and Arnu Pretorius. Should we be going MAD? a look at multi-agent debate strategies for LLMs. InProceedings of the 41st International Conference on Machine Learning, ICML’24...

    work page 2025

  10. [14]

    Jinwei Su, Qizhen Lan, Yinghui Xia, Lifan Sun, Weiyou Tian, Tianyu Shi, and Lewei He

    URL https://openreview.net/forum? id=4FWAwZtd2n. Jinwei Su, Qizhen Lan, Yinghui Xia, Lifan Sun, Weiyou Tian, Tianyu Shi, and Lewei He. Difficulty- Aware Agentic Orchestration for Query-Specific Multi-Agent Workflows. InProceedings of the ACM Web Conference 2026, WWW ’26, pages 2060–2070, New York, NY, USA,

    work page 2026

  11. [15]

    ISBN 979-8-4007-2307-0

    Association for Computing Machinery. ISBN 979-8-4007-2307-0. doi: 10.1145/3774904.3792240. URL https://doi.org/10.1145/3774904. 3792240. Dat Tran and Douwe Kiela. Single-Agent LLMs Outperform Multi-Agent Systems on Multi-Hop Reasoning Under Equal Thinking Token Budgets, April

    work page arXiv

  12. [16]

    arXiv:2604.02460 [cs.CL]

    URL http://arxiv.org/abs/2604.02460. arXiv:2604.02460 [cs.CL]. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models. InProceedings of the 36th International Conference on Neural Information Processing Systems, NIPS ’22, pa...

    Pith/arXiv arXiv

  13. [17]

    net/forum?id=YrycTjllL0

    URL https://openreview. net/forum?id=YrycTjllL0. Max Zimmer, Nico Pelleriti, Christophe Roux, and Sebastian Pokutta. The Agentic Researcher: A Practical Guide to AI-Assisted Research in Mathematics and Machine Learning. InICML 2026 Workshop: AI as a Tool for Mathematics, Computer Science, and Machine Learning,

    work page 2026

  14. [2022]

    URL https://proceedings.neurips.cc/paper_files/paper/ 2022/file/8bb0d291acd4acf06ef112099c16f326-Paper-Conference.pdf

    doi: 10.52202/068431-1613. URL https://proceedings.neurips.cc/paper_files/paper/ 2022/file/8bb0d291acd4acf06ef112099c16f326-Paper-Conference.pdf. junyou li, Qin Zhang, Yangbin Yu, QIANG FU, and Deheng Ye. More Agents Is All You Need. InTransactions on Machine Learning Research,

    work page doi:10.52202/068431-1613 2022

  15. [2023]

    arXiv:2312.11511 [cs] version:

    URL http://arxiv.org/abs/2312.11511. arXiv:2312.11511 [cs] version:

    Pith/arXiv arXiv

  16. [2024]

    Shangheng Du, Jiabao Zhao, Jinxin Shi, Zhentao Xie, Xin Jiang, Yanhong Bai, and Liang He

    URL https://openreview.net/forum?id=iNB4uoFQJb#discussion. Shangheng Du, Jiabao Zhao, Jinxin Shi, Zhentao Xie, Xin Jiang, Yanhong Bai, and Liang He. A Survey on the Optimization of Large Language Model-based Agents.ACM Comput. Surv., 58(9):223:1–223:37, February 2026a. ISSN 0360-0300. doi: 10.1145/3789261. URL https://dl.acm.org/doi/10.1145/3789261. Yilun...

    doi:10.1145/3789261

  17. [2025]

    Lingjiao Chen, Chi Zhang, Yeye He, Ion Stoica, Matei Zaharia, and James Zou

    URL https://proceedings.neurips.cc/paper_files/paper/2025/hash/b1041e52d3be19f0a9bc491657488e4a- Abstract-Datasets_and_Benchmarks_Track.html. Lingjiao Chen, Chi Zhang, Yeye He, Ion Stoica, Matei Zaharia, and James Zou. The Price Reversal Phenomenon: When Cheaper Reasoning Models Cost More, May 2026a. URL http://arxiv.org/abs/2603. 23971. arXiv:2603.23971 ...

    Pith/arXiv arXiv 2025

  18. [2026]

    MAS-PromptBench: When Does Prompt Optimization Improve Multi-Agent LLM Systems?

    URL http://arxiv.org/abs/2606.23664. arXiv:2606.23664 [cs.LG] version:

    work page internal anchor Pith review Pith/arXiv arXiv

This paper was first reviewed by deepseek-v4-flash on August 5, 2026.