Pith. sign in

REVIEW 3 major objections 5 minor 1 cited by

MASPOB: Bandit-Based Prompt Optimization for Multi-Agent Systems with Graph Neural Networks

T0 review · 3 major / 5 minor · reviewed 2026-08-02 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Multi-agent system prompts can be optimized within 50 evaluations by a bandit search whose surrogate reads the workflow's graph, and this beats existing single-agent and multi-agent prompt optimizers across six LLM benchmarks.

desk verdict Useful new combination of GNN surrogate and bandit search for prompt optimization in frozen-topology MAS, but the headline overclaims and the comparison against MIPRO is confounded by candidate-prompt domain. read the letter →

arxiv 2603.02630 v2 pith:TUDUROAJ submitted 2026-03-03 cs.LG cs.AI

Zhi Hong , Qian Zhang , Jiahang Sun , Zhiwei Shang , Mingze Kong , Xiangyi Wang , Yao Shu , Zhongxiang Dai This is my paper · ORCID
classification cs.LGcs.AI MSC 68T0768T05
keywords promptoptimizationmulti-agentsystemslargelanguagemodelscontextualbanditsgraphneuralnetworksupperconfidenceboundcoordinateascentsampleefficiency
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Many deployed multi-agent LLM systems cannot have their workflows restructured, because the pipelines are compliance- or expert-vetted; the only lever left is the text of each agent's prompt. MASPOB claims that this lever is enough: it treats prompt search as a budgeted black-box optimization and finds well-coordinated prompt combinations in only 50 end-to-end runs. Its core claim is that the search works by modeling the workflow's topology—using a graph neural network surrogate—so that changing one agent's prompt is evaluated with respect to how it changes what downstream agents receive, while a bandit-style upper-confidence-bound rule decides which combinations to try next. Across six benchmarks (question answering, code generation, and mathematical reasoning) the paper reports that MASPOB outperforms strong single- and multi-agent baselines under the same evaluation budget, averaging 80.58%. The paper is explicit that this result is conditional on the candidate prompt pool: the optimizer selects among pre-generated variants, so it cannot recover a prompt that was never drafted.

What carries the argument

Three components carry the argument. (1) A graph neural network (GNN) surrogate: the workflow is a DAG; each agent is a node whose feature is the embedding of its current prompt, and attention-based message passing lets the surrogate predict how a prompt change ripples downstream. (2) A linear upper-confidence-bound (UCB) rule: an information matrix accumulates the embeddings of evaluated combinations; the term sqrt(φ(c)ᵀ M⁻¹ φ(c)) estimates uncertainty, and the acquisition score adds this to the GNN's predicted score, balancing exploitation and exploration. (3) Coordinate ascent: starting from the incumbent best combination, each agent's prompt is greedily replaced with the one maximizing U

What would settle it

Run MASPOB on the same budgets but with a deliberately degraded candidate pool (e.g., 20 near-duplicate paraphrases of one prompt) and compare to random selection from that pool: if random search matches or beats MASPOB, the bandit/GNN machinery is not the source of the reported gains. The complementary test is to give the same 50-evaluation budget to a simple evolutionary search over an identical candidate pool: if the simple baseline matches MASPOB's scores, the topology-aware surrogate and UCB exploration contribute nothing beyond ordinary search.

Watch

Extended reading notes

Core claim

The paper's central discovery claim is that the bottleneck in optimizing fixed-topology multi-agent systems is not the prompt search per se, but the way the search handles inter-agent coupling—and that this coupling can be modeled with a graph. MASPOB represents each agent as a node in the workflow DAG, feeds the candidate prompts' embeddings through a graph-neural-network surrogate to predict system performance, and adds a linear upper-confidence-bound bonus that grows with how unexplored a prompt combination is in embedding space. A coordinate-ascent loop then updates one agent's prompt at a time against the UCB score, cutting the per-round search from exponential to linear in the number o

Load-bearing premise

MASPOB can only pick from fixed, pre-generated candidate prompts per agent, so if the best prompt for an agent was never drafted, the search cannot find it and all reported gains are bounded by the quality and diversity of that initial candidate pool (a limitation the paper states explicitly in Section 4 and Appendix A.3).

Editorial extensions

If this is right

  • With a budget of 50 end-to-end evaluations, MASPOB improves average test accuracy by about 12 percentage points over plain prompting and about 2 points over the strongest multi-agent prompt baseline.
  • The improvements appear across all six benchmarks—QA, code generation, and math—suggesting the benefit comes from coordination rather than task-specific prompt content.
  • Removing the graph surrogate costs about 2.3 average points, indicating that topology-aware modeling is a measurable source of the gain, not a cosmetic addition.
  • Coordinate ascent matches exhaustive search within roughly 0.3–0.5 points but runs 98–99.8% faster, so the search for good prompt combinations is computationally practical.
  • The benefit transfers to a different backbone LLM and to an alternative, independently generated candidate pool, which the paper interprets as evidence that the gains come from the optimization procedure rather than from a single model or a single prompt-domain recipe.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because MASPOB only selects among pre-written candidates, its ceiling is set by the prompt generator; a natural next step is to let the bandit's uncertainty signal trigger the drafting of new variants in unexplored regions, converting selection into closed-loop generation.
  • The reported scaling suggests the GNN surrogate's advantage should grow with workflow size and coupling: a testable prediction is that on workflows with more agents or with feedback edges (currently excluded by the DAG assumption), a topology-aware surrogate will separate from structure-blind baselines by a larger margin than the roughly 2.3 points seen here.
  • The near-tie between coordinate ascent and global search suggests the UCB bonus itself may be supplying the global exploration that makes coordinate-wise greedy updates safe; ablating the exploration coefficient or removing the bonus on a fixed budget would test whether the safety comes from the bonus or from the smoothness of the performance landscape.
  • For practitioners in regulated settings, the practical implication is that prompt tuning can substitute for workflow restructuring up to a point: the paper's protocol preserves the audited topology and still delivers gains, which aligns with deployment constraints where re-validation of the workflow is expensive or forbidden.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes MASPOB, a prompt optimizer for multi-agent systems (MAS) with a fixed workflow topology. The method combines a GAT-based surrogate, a LinUCB-style exploration bonus, and coordinate ascent to search a discrete Cartesian product of per-agent prompt variants under a tight evaluation budget (50 end-to-end executions). The authors evaluate on six benchmarks spanning QA, code generation, and mathematical reasoning, comparing against single-agent baselines (IO, CoT, ReAct, PromptBreeder, Instinct) and multi-agent methods (AFlow, MIPRO). They report an average score of 80.58% and claim consistent state-of-the-art performance. Ablations investigate the GNN, uncertainty estimation, warm-up, embedding model, coordinate ascent, and exploration coefficient.

Significance. If the results hold, MASPOB addresses a practically important and under-studied problem: sample-efficient prompt optimization for already-validated, frozen MAS workflows. The combination of a topology-aware GNN surrogate with a linear UCB bonus is well motivated, and the paper provides a fairly extensive experimental suite with a fixed 50-evaluation budget, ablations on several design components, robustness checks across LLMs and prompt domains, and a complexity comparison against exhaustive search. The public code link and detailed hyperparameters (Appendix A.3) support reproducibility. The main contribution is empirical rather than theoretical: no regret or convergence guarantees are given, but the algorithmic structure is sensible and the experimental setup is largely careful. The principal weakness is that the headline comparison against the closest prompt-optimization baseline, MIPRO, is not fully controlled with respect to how the candidate prompt pool is constructed, and a stated claim of 'best on every benchmark' is contradicted by a tie in Table 1.

major comments (3)
  1. [§4.2, Table 1, Abstract] The sentence 'MASPOB achieves the best result on every benchmark' is not supported by Table 1. On MBPP, MASPOB and MIPRO both report 80.65, i.e., a tie, and the table's bolding gives MASPOB sole credit. The abstract's 'consistently outperforming' is likewise too strong. Please revise to 'matches or outperforms' or otherwise qualify the claim, and correct the bolding.
  2. [§4.2, Table 5, Appendix A.3] The comparison between MASPOB and MIPRO is confounded with prompt-domain construction. MASPOB builds its candidate pool via 20 GPT-4o-mini style-controlled paraphrases per agent (Appendix A.3), while MIPRO uses its own data/program/fewshot/tip-aware strategies. Table 5 only swaps MIPRO's domain into MASPOB, showing similar scores; it never runs MIPRO on MASPOB's domain. Section 4 also concedes that 'the quality and diversity of candidate prompts can still affect absolute performance.' Therefore the reported 1.71-point average advantage over MIPRO could be partly or wholly due to the candidate pool rather than the GNN/UCB/coordinate-ascent selection. Please run MIPRO on the MASPOB candidate pool (or otherwise hold the candidate domain fixed across optimizers) and report both directions before claiming that the gains 'mainly come from topology-aware contextual-bandit optimization.'
  3. [§6 Related Work, Tables 1-4] The paper identifies MAPRO (Zhang et al., 2025c) as 'the closest prior work to ours' and as a principled multi-agent prompt optimizer, yet MAPRO is never included as a baseline. For a state-of-the-art claim, omitting the closest competitor is a significant gap. Please add MAPRO to the main comparisons if its code/API allows, or provide a concrete reason (e.g., incompatibility with the fixed-budget protocol) why it cannot be included.
minor comments (5)
  1. [Table 5 caption] The caption has a typo: 'We report mean accuracy (standard deviation over three runs' is missing the '±' and an opening bracket. Also, Table 5 reports only DROP and MATH, while the text in §4.2 says 'as shown in Table 5' without noting the limited coverage; please state the scope explicitly.
  2. [§3.2, Algorithm 1, Table 8] The hyperparameter table lists a 'Fisher matrix update coefficient' of 10, but Algorithm 1 and Eq. (10) update the information matrix as M ← M + Φ(c)Φ(c)⊤ with no coefficient. Either the algorithm description is missing a scaling factor or the table entry is unused. Please clarify.
  3. [§4.1, Metrics] MATH is referred to as 'MATHlv5*' in the metrics paragraph, which is inconsistent with the dataset name 'MATH' elsewhere. Please unify terminology.
  4. [§4.2, Figure 3] The convergence figure reports test accuracy at checkpoints every 5 rounds and validation as a binned average, but the caption does not define how the test checkpoints are averaged (the text says 'evaluated and averaged over three runs'). Please state whether the three runs are the three final test repetitions or separate optimization runs.
  5. [§4.2, Table 1] The 12.02% average improvement over IO is emphasized as a headline result, but IO is a single-LLM-call baseline while MASPOB uses a multi-agent workflow with multiple LLM calls. The evaluation budget is matched in number of full-workflow executions, not in inference cost. Please add a sentence clarifying this distinction so readers do not interpret the gain as being achieved at equal total LLM inference cost.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: MASPOB's GNN/UCB surrogate is trained on observed validation scores and evaluated on held-out test benchmarks; self-citations to neural-bandit prior work are not load-bearing.

full rationale

MASPOB's central claim is an empirical comparison on six external benchmarks. The GNN surrogate is trained on validation-set scores obtained by end-to-end MAS executions (Eq. 1, Algorithm 1 lines 5-9 and 22-25), and the final prompt combination is selected by validation performance and then evaluated on a held-out test split (Section 2 and Appendix A.1); the test labels are never used to fit the surrogate or the UCB information matrix. The LinUCB uncertainty term (Eqs. 11-12) is a standard exploration bonus over the same fitted representation, not a renamed version of the objective being predicted. Coordinate ascent (Eq. 13) only reduces acquisition-search cost, and Table 6 shows it tracks global search on the same acquisition function. The paper cites prior work by its own authors (e.g., Lin et al. 2023; Wu et al. 2024b; Kong et al. 2025) for the bandit formulation and for the Instinct baseline, but no load-bearing step invokes those papers as an external fact that forces the result; the 'NeuralLinear-style' decomposition in Appendix B is presented as a design choice and is validated against a neural-uncertainty ablation (Table 9). The prompt-domain robustness experiment (Table 5) only swaps MIPRO's candidate domain into MASPOB, so the MIPRO comparison may be confounded by candidate-prompt quality, but that is an experimental-design/validity concern, not a definitional or self-citational reduction; Section 4 explicitly concedes that 'the quality and diversity of candidate prompts can still affect absolute performance, since the optimizer can only select from the provided prompt domain.' Overall, no equation or fitted parameter is constructed so that the reported benchmark improvement is true by definition.

Assumptions & free parameters 5 free parameters · 6 assumptions · 0 invented entities

No new entities are introduced; the contribution is a composition of GAT, LinUCB, and coordinate ascent. The load-bearing assumptions are statistical and modeling assumptions about the surrogate, the candidate prompt domain, and the small validation splits.

free parameters (5)
  • Exploration coefficient α = 0.2
    Controls UCB bonus; selected via sensitivity sweep in Table 14, with no separate held-out tuning budget described.
  • Regularization coefficient λ = 1.0
    Initializes the information matrix M = λI; listed in Table 8 and not sensitivity-tested in the paper.
  • Warm-up rounds T0 = 5
    Number of random evaluations before GNN training; selected via sensitivity analysis in Table 10.
  • Fisher matrix update coefficient = 10 (unexplained)
    Listed in Table 8 but absent from Eq. (10), which shows M ← M + ΦΦ^T; appears to be an implementation-specific scaling not described in the text.
  • Prompt variants per agent = 20
    Candidate-set size is chosen by hand; the optimizer cannot select prompts outside this set, making this a load-bearing experimental-design parameter.
assumptions (6)
  • domain assumption MAS inter-agent information flow is a static DAG
    Section 2 states the DAG assumption applies only to inter-agent information flow within a single execution; cross-agent feedback cycles are excluded. The method and all experiments presume this topology.
  • domain assumption Small validation splits are reliable proxies for test performance
    Appendix A.1 uses validation sets as small as 33 (HumanEval) and 86 (MBPP) prompts; the optimizer selects prompts on these splits and the paper reports no error bars on the validation signal.
  • ad hoc to paper GAT surrogate trained on ≤50 observations provides useful performance predictions
    Section 3.1 and Algorithm 1 rely on a GAT trained on 5 warm-up plus up to 45 evaluated combinations; no generalization guarantee is provided.
  • ad hoc to paper LinUCB uncertainty in combined prompt-embedding space is a valid exploration signal
    Section 3.2 assumes a linear payoff structure in the concatenated embedding space for MAS performance, which is not established for the true performance function.
  • ad hoc to paper Coordinate ascent on a non-concave UCB acquisition function finds near-optimal combinations
    Section 3.3 uses coordinate ascent with no convergence guarantee; only two empirical comparisons against global search (Table 6) support the approximation.
  • domain assumption Pretrained prompt embeddings preserve task-relevant semantic differences
    Appendix A.3 uses Qwen3-8B embeddings as node features; Table 11 shows performance varies with embedding backbone, confirming dependence on this assumption.

how reviews work

0 comments
Cite this review

Pith. "Pith review of MASPOB: Bandit-Based Prompt Optimization for Multi-Agent Systems with Graph Neural Networks." pith.science (2026). https://pith.science/paper/TUDUROAJ

@misc{pith2026260302630,
  author       = {Pith},
  title        = {Pith review of: MASPOB: Bandit-Based Prompt Optimization for Multi-Agent Systems with Graph Neural Networks},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/TUDUROAJ}},
  note         = {Machine review of arXiv:2603.02630}
}
read the original abstract

Large Language Models (LLMs) have achieved great success in many real-world applications, especially the one serving as the cognitive backbone of Multi-Agent Systems (MAS) to orchestrate complex workflows in practice. Since many deployment scenarios preclude MAS workflow modifications and its performance is highly sensitive to the input prompts, prompt optimization emerges as a more natural approach to improve its performance. However, real-world prompt optimization for MAS is impeded by three key challenges: (1) the need of sample efficiency due to prohibitive evaluation costs, (2) topology-induced coupling among prompts, and (3) the combinatorial explosion of the search space. To address these challenges, we introduce MASPOB (Multi-Agent System Prompt Optimization via Bandits), a novel sample-efficient framework based on bandits. By leveraging Upper Confidence Bound (UCB) to quantify uncertainty, the bandit framework balances exploration and exploitation, maximizing gains within a strictly limited budget. To handle topology-induced coupling, MASPOB integrates Graph Neural Networks (GNNs) to capture structural priors, learning topology-aware representations of prompt semantics. Furthermore, it employs coordinate ascent to decompose the optimization into univariate sub-problems, reducing search complexity from exponential to linear. Extensive experiments across diverse benchmarks demonstrate that MASPOB achieves state-of-the-art performance, consistently outperforming existing baselines.

Figures

Figures reproduced from arXiv: 2603.02630 by the authors.

Figure 1
Figure 1. The MASPOB framework. (1) Initialization: Constructs agent topology and generates prompt embeddings. (2) Search: Selects optimal prompts via Coordinate Ascent, balancing exploitation (GNN prediction) and exploration (Linear UCB uncertainty). (3) Update: Refines the GNN model and information matrix using execution feedback. Despite this motivation, existing prompt optimizers do not fully address the above setting. Si… view at source ↗
Figure 2
Figure 2. Performance evaluation of prompt optimization methods. (a) Detailed comparison across six diverse benchmarks including question answering (HotpotQA, DROP), code generation (HumanEval, MBPP), and mathematical reasoning (GSM8K, MATH). (b) Overall ranking based on average performance. MASPOB demonstrates superior performance with an average improvement of 12.02% over the IO baseline. 2. Problem Setting We study prompt … view at source ↗
Figure 3
Figure 3. Optimization convergence on validation and test sets. The curves show the average validation accuracy, computed by averaging over every five rounds, and the test accuracy at rounds 5, 10, . . . , 50. For each selected test combination, the accuracy at these rounds is evaluated and averaged over three runs. math) and structured-output tasks (e.g., code generation). This suggests that the benefit is not limited to a s… view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Performance and runtime comparison between coordi￾nate ascent and global search. The figure illustrates optimization trajectories and time costs on selected benchmarks. Most of these methods target single-agent settings. For multi-stage pipelines, MIPRO (Opsahl-Ong et …
Figure 5
Figure 5. Figure 5: Uncertainty convergence over optimization rounds. Linear uncertainty decreases by 71.68%, whereas neural uncertainty decreases by 22.48% within 45 rounds, suggesting that neural uncertainty may require more exploration to reach comparable confidence. C. Additional Robu…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. MAS-PromptBench: When Does Prompt Optimization Improve Multi-Agent LLM Systems?

    cs.LG 2026-06 unverdicted novelty 6.0 of 10

    A new benchmark study finds that prompt optimization can deliver significant gains in multi-agent LLM systems but its effectiveness varies strongly with task, workflow, communication protocol, and team size.

Reference graph

Works this paper leans on

59 extracted references · 31 linked inside Pith · cited by 1 Pith paper

  1. [1]

    Improved algorithms for linear stochastic bandits

    Abbasi-Yadkori, Y., P \'a l, D., and Szepesv \'a ri, C. Improved algorithms for linear stochastic bandits. Advances in neural information processing systems, 24, 2011

    2011

  2. [2]

    Program synthesis with large language models

    Austin, J., Odena, A., Nye, M., Bosma, M., Michalewski, H., Dohan, D., Jiang, E., Cai, C., Terry, M., Le, Q., et al. Program synthesis with large language models. arXiv preprint arXiv:2108.07732, 2021

    arXiv 2021

  3. [3]

    and Travis, J

    Bodnari, A. and Travis, J. Scaling enterprise ai in healthcare: the role of governance in risk mitigation frameworks. npj Digital Medicine, 8 0 (1): 0 272, 2025

    2025

  4. [4]

    D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al

    Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. Language models are few-shot learners. Advances in neural information processing systems, 33: 0 1877--1901, 2020

    1901

  5. [5]

    Instructzero: Efficient instruction optimization for black-box large language models

    Chen, L., Chen, J., Goldstein, T., Huang, H., and Zhou, T. Instructzero: Efficient instruction optimization for black-box large language models. arXiv preprint arXiv:2306.03082, 2023 a

    arXiv 2023

  6. [6]

    Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. D. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021

    arXiv 2021

  7. [7]

    Agentverse: Facilitating multi-agent collaboration and exploring emergent behaviors

    Chen, W., Su, Y., Zuo, J., Yang, C., Yuan, C., Chan, C.-M., Yu, H., Lu, Y., Hung, Y.-H., Qian, C., et al. Agentverse: Facilitating multi-agent collaboration and exploring emergent behaviors. In The Twelfth International Conference on Learning Representations, 2023 b

    2023

  8. [8]

    Contextual bandits with linear payoff functions

    Chu, W., Li, L., Reyzin, L., and Schapire, R. Contextual bandits with linear payoff functions. In Proceedings of the fourteenth international conference on artificial intelligence and statistics, pp.\ 208--214. JMLR Workshop and Conference Proceedings, 2011

    2011

Show all 59 references
  1. [9]

    Training verifiers to solve math word problems

    Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021

    2021 arXiv

  2. [10]

    Rlprompt: Optimizing discrete text prompts with reinforcement learning

    Deng, M., Wang, J., Hsieh, C.-P., Wang, Y., Guo, H., Shu, T., Song, M., Xing, E., and Hu, Z. Rlprompt: Optimizing discrete text prompts with reinforcement learning. In Proceedings of the 2022 conference on empirical methods in natural language processing, pp.\ 3369--3391, 2022

    2022

  3. [11]

    Drop: A reading comprehension benchmark requiring discrete reasoning over paragraphs

    Dua, D., Wang, Y., Dasigi, P., Stanovsky, G., Singh, S., and Gardner, M. Drop: A reading comprehension benchmark requiring discrete reasoning over paragraphs. arXiv preprint arXiv:1903.00161, 2019

    1903 arXiv

  4. [12]

    Promptbreeder: Self-referential self-improvement via prompt evolution

    Fernando, C., Banarse, D., Michalewski, H., Osindero, S., and Rockt \"a schel, T. Promptbreeder: Self-referential self-improvement via prompt evolution. arXiv preprint arXiv:2309.16797, 2023

    2023 arXiv

  5. [13]

    Connecting large language models with evolutionary algorithms yields powerful prompt optimizers

    Guo, Q., Wang, R., Guo, J., Li, B., Song, K., Tan, X., Liu, G., Bian, J., and Yang, Y. Connecting large language models with evolutionary algorithms yields powerful prompt optimizers. arXiv preprint arXiv:2309.08532, 2023

    2023 arXiv

  6. [14]

    Measuring mathematical problem solving with the math dataset

    Hendrycks, D., Burns, C., Kadavath, S., Arora, A., Basart, S., Tang, E., Song, D., and Steinhardt, J. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874, 2021

    2021 arXiv

  7. [15]

    Hong, S., Zhuge, M., Chen, J., Zheng, X., Cheng, Y., Wang, J., Zhang, C., Wang, Z., Yau, S. K. S., Lin, Z., et al. Metagpt: Meta programming for a multi-agent collaborative framework. In The twelfth international conference on learning representations, 2023

    2023

  8. [16]

    Automated design of agentic systems

    Hu, S., Lu, C., and Clune, J. Automated design of agentic systems. arXiv preprint arXiv:2408.08435, 2024

    2024 arXiv

  9. [17]

    P., Perelman, A., Ramesh, A., Clark, A., Ostrow, A., Welihinda, A., Hayes, A., Radford, A., et al

    Hurst, A., Lerer, A., Goucher, A. P., Perelman, A., Ramesh, A., Clark, A., Ostrow, A., Welihinda, A., Hayes, A., Radford, A., et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276, 2024

    2024 arXiv

  10. [18]

    T., Moazam, H., et al

    Khattab, O., Singhvi, A., Maheshwari, P., Zhang, Z., Santhanam, K., Vardhamanan, S., Haq, S., Sharma, A., Joshi, T. T., Moazam, H., et al. Dspy: Compiling declarative language model calls into self-improving pipelines. arXiv preprint arXiv:2310.03714, 2023

    2023 arXiv

  11. [19]

    S., Reid, M., Matsuo, Y., and Iwasawa, Y

    Kojima, T., Gu, S. S., Reid, M., Matsuo, Y., and Iwasawa, Y. Large language models are zero-shot reasoners. Advances in neural information processing systems, 35: 0 22199--22213, 2022

    2022

  12. [20]

    Meta-prompt optimization for llm-based sequential decision making

    Kong, M., Wang, Z., Shu, Y., and Dai, Z. Meta-prompt optimization for llm-based sequential decision making. arXiv preprint arXiv:2502.00728, 2025

    2025 arXiv

  13. [21]

    Camel: Communicative agents for" mind" exploration of large language model society

    Li, G., Hammoud, H., Itani, H., Khizbullin, D., and Ghanem, B. Camel: Communicative agents for" mind" exploration of large language model society. Advances in Neural Information Processing Systems, 36: 0 51991--52008, 2023

    2023

  14. [22]

    Li, L., Chu, W., Langford, J., and Schapire, R. E. A contextual-bandit approach to personalized news article recommendation. In Proceedings of the 19th international conference on World wide web, pp.\ 661--670, 2010

    2010

  15. [23]

    Combinatorial optimization with graph convolutional networks and guided tree search

    Li, Z., Chen, Q., and Koltun, V. Combinatorial optimization with graph convolutional networks and guided tree search. Advances in neural information processing systems, 31, 2018

    2018

  16. [24]

    Lin, X., Wu, Z., Dai, Z., Hu, W., Shu, Y., Ng, S.-K., Jaillet, P., and Low, B. K. H. Use your instinct: Instruction optimization using neural bandits coupled with transformers. arXiv preprint arXiv:2310.02905, 2023

    2023 arXiv

  17. [25]

    Lin, X., Dai, Z., Verma, A., Ng, S.-K., Jaillet, P., and Low, B. K. H. Prompt optimization with human feedback. arXiv preprint arXiv:2405.17346, 2024

    2024 arXiv

  18. [26]

    Dynamic llm-agent network: An llm-agent collaboration framework with agent team optimization

    Liu, Z., Zhang, Y., Li, P., Liu, Y., and Yang, D. Dynamic llm-agent network: An llm-agent collaboration framework with agent team optimization. arXiv preprint arXiv:2310.02170, 2023

    2023 arXiv

  19. [27]

    Sop-bench: Complex industrial sops for evaluating llm agents

    Nandi, S., Datta, A., Vichare, N., Bhattacharya, I., Raja, H., Xu, J., Ray, S., Carenini, G., Srivastava, A., Chan, A., et al. Sop-bench: Complex industrial sops for evaluating llm agents. arXiv preprint arXiv:2506.08119, 2025

    2025

  20. [29]

    J., Purtell, J., Broman, D., Potts, C., Zaharia, M., and Khattab, O

    Opsahl-Ong, K., Ryan, M. J., Purtell, J., Broman, D., Potts, C., Zaharia, M., and Khattab, O. Optimizing instructions and demonstrations for multi-stage language model programs. arXiv preprint arXiv:2406.11695, 2024 b

    2024 arXiv

  21. [30]

    Flow-of-action: Sop enhanced llm-based multi-agent system for root cause analysis

    Pei, C., Wang, Z., Liu, F., Li, Z., Liu, Y., He, X., Kang, R., Zhang, T., Chen, J., Li, J., et al. Flow-of-action: Sop enhanced llm-based multi-agent system for root cause analysis. In Companion Proceedings of the ACM on Web Conference 2025, pp.\ 422--431, 2025

    2025

  22. [31]

    Grips: Gradient-free, edit-based instruction search for prompting large language models

    Prasad, A., Hase, P., Zhou, X., and Bansal, M. Grips: Gradient-free, edit-based instruction search for prompting large language models. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pp.\ 3845--3864, 2023

    2023

  23. [32]

    gradient descent

    Pryzant, R., Iter, D., Li, J., Lee, Y. T., Zhu, C., and Zeng, M. Automatic prompt optimization with" gradient descent" and beam search. arXiv preprint arXiv:2305.03495, 2023

    2023 arXiv

  24. [33]

    Chatdev: Communicative agents for software development

    Qian, C., Liu, W., Liu, H., Chen, N., Dang, Y., Li, J., Yang, C., Chen, W., Su, Y., Cong, X., et al. Chatdev: Communicative agents for software development. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp.\...

    2024

  25. [34]

    Verifai: verified generative ai

    Tang, N., Yang, C., Fan, J., Cao, L., Luo, Y., and Halevy, A. Verifai: verified generative ai. arXiv preprint arXiv:2307.02796, 2023

    2023 arXiv

  26. [35]

    Graph attention networks

    Veli c kovi \'c , P., Cucurull, G., Casanova, A., Romero, A., Lio, P., and Bengio, Y. Graph attention networks. arXiv preprint arXiv:1710.10903, 2017

    2017 arXiv

  27. [36]

    Voyager: An open-ended embodied agent with large language models

    Wang, G., Xie, Y., Jiang, Y., Mandlekar, A., Xiao, C., Zhu, Y., Fan, L., and Anandkumar, A. Voyager: An open-ended embodied agent with large language models. arXiv preprint arXiv:2305.16291, 2023 a

    2023 arXiv

  28. [37]

    Self-consistency improves chain of thought reasoning in language models, 2023 b

    Wang, X., Wei, J., Schuurmans, D., Le, Q., Chi, E., Narang, S., Chowdhery, A., and Zhou, D. Self-consistency improves chain of thought reasoning in language models, 2023 b . URL https://arxiv.org/abs/2203.11171

    2023 arXiv

  29. [38]

    V., Zhou, D., et al

    Wei, J., Wang, X., Schuurmans, D., Bosma, M., Xia, F., Chi, E., Le, Q. V., Zhou, D., et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35: 0 24824--24837, 2022

    2022

  30. [39]

    J., Nguyen, H

    Wells, B. J., Nguyen, H. M., McWilliams, A., Pallini, M., Bovi, A., Kuzma, A., Kramer, J., Chou, S.-H., Hetherington, T., Corn, P., et al. A practical framework for appropriate implementation and review of artificial intelligence (fair-ai) in healthcare. NPJ digital medicine, ...

    2025

  31. [40]

    Medreason: Eliciting factual medical reasoning steps in llms via knowledge graphs

    Wu, J., Deng, W., Li, X., Liu, S., Mi, T., Peng, Y., Xu, Z., Liu, Y., Cho, H., Choi, C.-I., et al. Medreason: Eliciting factual medical reasoning steps in llms via knowledge graphs. arXiv preprint arXiv:2504.00993, 2025

    2025 arXiv

  32. [41]

    Autogen: Enabling next-gen llm applications via multi-agent conversations

    Wu, Q., Bansal, G., Zhang, J., Wu, Y., Li, B., Zhu, E., Jiang, L., Zhang, X., Zhang, S., Liu, J., et al. Autogen: Enabling next-gen llm applications via multi-agent conversations. In First Conference on Language Modeling, 2024 a

    2024

  33. [42]

    Wu, Z., Lin, X., Dai, Z., Hu, W., Shu, Y., Ng, S.-K., Jaillet, P., and Low, B. K. H. Prompt optimization with ease? efficient ordering-aware automated selection of exemplars. Advances in Neural Information Processing Systems, 37: 0 122706--122740, 2024 b

    2024

  34. [43]

    Qwen3 technical report

    Yang, A., Li, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Gao, C., Huang, C., Lv, C., et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025

    2025 arXiv

  35. [44]

    V., Zhou, D., and Chen, X

    Yang, C., Wang, X., Lu, Y., Liu, H., Le, Q. V., Zhou, D., and Chen, X. Large language models as optimizers. In The Twelfth International Conference on Learning Representations, 2023

    2023

  36. [45]

    Finrobot: An open-source ai agent platform for financial applications using large language models

    Yang, H., Zhang, B., Wang, N., Guo, C., Zhang, X., Lin, L., Wang, J., Zhou, T., Guan, M., Zhang, R., et al. Finrobot: An open-source ai agent platform for financial applications using large language models. arXiv preprint arXiv:2405.14767, 2024

    2024 arXiv

  37. [46]

    Yang, Z., Qi, P., Zhang, S., Bengio, Y., Cohen, W., Salakhutdinov, R., and Manning, C. D. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. In Proceedings of the 2018 conference on empirical methods in natural language processing, pp.\ 2369--2380, 2018

    2018

  38. [47]

    R., and Cao, Y

    Yao, S., Zhao, J., Yu, D., Du, N., Shafran, I., Narasimhan, K. R., and Cao, Y. React: Synergizing reasoning and acting in language models. In The eleventh international conference on learning representations, 2022

    2022

  39. [48]

    Tree of thoughts: Deliberate problem solving with large language models

    Yao, S., Yu, D., Zhao, J., Shafran, I., Griffiths, T., Cao, Y., and Narasimhan, K. Tree of thoughts: Deliberate problem solving with large language models. Advances in neural information processing systems, 36: 0 11809--11822, 2023 a

    2023

  40. [49]

    React: Synergizing reasoning and acting in language models, 2023 b

    Yao, S., Zhao, J., Yu, D., Du, N., Shafran, I., Narasimhan, K., and Cao, Y. React: Synergizing reasoning and acting in language models, 2023 b . URL https://arxiv.org/abs/2210.03629

    2023 arXiv

  41. [50]

    Sop-agent: Empower general purpose ai agent with domain-specific sops

    Ye, A., Ma, Q., Chen, J., Li, M., Li, T., Liu, F., Mai, S., Lu, M., Bao, H., and You, Y. Sop-agent: Empower general purpose ai agent with domain-specific sops. arXiv preprint arXiv:2501.09316, 2025

    2025 arXiv

  42. [51]

    Thought propagation: An analogical approach to complex reasoning with large language models

    Yu, J., He, R., and Ying, R. Thought propagation: An analogical approach to complex reasoning with large language models. arXiv preprint arXiv:2310.03965, 2023

    2023 arXiv

  43. [52]

    differentiation

    Yuksekgonul, M., Bianchi, F., Boen, J., Liu, S., Huang, Z., Guestrin, C., and Zou, J. Textgrad: Automatic" differentiation" via text. arXiv preprint arXiv:2406.07496, 2024

    2024 arXiv

  44. [53]

    Aflow: Automating agentic workflow generation

    Zhang, J., Xiang, J., Yu, Z., Teng, F., Chen, X., Chen, J., Zhuge, M., Cheng, X., Hong, S., Wang, J., et al. Aflow: Automating agentic workflow generation. arXiv preprint arXiv:2410.10762, 2024

    2024 arXiv

  45. [54]

    Gnns as predictors of agentic workflow performances

    Zhang, Y., Hou, Y., Tang, B., Chen, S., Zhang, M., Dong, X., and Chen, S. Gnns as predictors of agentic workflow performances. arXiv preprint arXiv:2503.11301, 2025 a

    2025 arXiv

  46. [55]

    Qwen3 embedding: Advancing text embedding and reranking through foundation models

    Zhang, Y., Li, M., Long, D., Zhang, X., Lin, H., Yang, B., Xie, P., Yang, A., Liu, D., Lin, J., et al. Qwen3 embedding: Advancing text embedding and reranking through foundation models. arXiv preprint arXiv:2506.05176, 2025 b

    2025 arXiv

  47. [56]

    Mapro: Recasting multi-agent prompt optimization as maximum a posteriori inference, 2025 c

    Zhang, Z., Ge, L., Li, H., Zhu, W., Zhang, C., and Ye, Y. Mapro: Recasting multi-agent prompt optimization as maximum a posteriori inference, 2025 c . URL https://arxiv.org/abs/2510.07475

    2025

  48. [57]

    Neural contextual bandits with ucb-based exploration

    Zhou, D., Li, L., and Gu, Q. Neural contextual bandits with ucb-based exploration. In International conference on machine learning, pp.\ 11492--11502. PMLR, 2020

    2020

  49. [58]

    I., Han, Z., Paster, K., Pitis, S., Chan, H., and Ba, J

    Zhou, Y., Muresanu, A. I., Han, Z., Paster, K., Pitis, S., Chan, H., and Ba, J. Large language models are human-level prompt engineers. In The eleventh international conference on learning representations, 2022

    2022

  50. [59]

    Gptswarm: Language agents as optimizable graphs

    Zhuge, M., Wang, W., Kirsch, L., Faccio, F., Khizbullin, D., and Schmidhuber, J. Gptswarm: Language agents as optimizable graphs. In Forty-first International Conference on Machine Learning, 2024

    2024

  51. [60]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 2, 2026 · model on record in the stance chip above.