EvoFlow: Evolving Diverse Agentic Workflows On The Fly

Guancheng Wan; Guibin Zhang; Heng Chang; Hong Cheng; Kaijie Chen; Kun Wang; Lei Bai; Shuyue Hu

REVIEW 5 major objections 5 minor 12 cited by

EvoFlow: Evolving Diverse Agentic Workflows On The Fly

T0 review · 5 major / 5 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims the right unit of agentic design is a Pareto population of workflows, not a single workflow.

desk verdict A genuinely new multi-objective twist on agentic workflow search, with evidence that needs major cleanup before the numbers can be trusted. read the letter →

arxiv 2502.07373 v1 pith:IFMYWX5L submitted 2025-02-11 cs.LG cs.CLcs.MAcs.NE

Guibin Zhang , Kaijie Chen , Guancheng Wan , Heng Chang , Hong Cheng , Kun Wang , Shuyue Hu , Lei Bai This is my paper · ORCID

classification cs.LGcs.CLcs.MAcs.NE

keywords agenticworkflowsmulti-agentsystemsevolutionaryalgorithmnichingselectionmulti-objectiveoptimizationParetofrontLLMheterogeneityautomatedworkflowdesign

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

EvoFlow claims that the right object to optimize in multi-agent LLM systems is not a single workflow but a population of workflows spread along a cost-performance Pareto front. It formulates agentic workflow search as a multi-objective problem—maximize task accuracy, minimize inference cost—and solves it with a niching evolutionary algorithm whose individuals are executable workflow graphs made of operator nodes such as CoT, debate, ensemble, and self-reflexion. The population is evolved on incoming queries via tag-based parent retrieval, LLM-facilitated crossover, three mutation operators (LLM, prompt, and operator mutation), and niche-based elimination, so that simple queries get cheap workflows and hard queries get expensive multi-turn ones. Across six benchmarks the paper reports average gains of 1.23%–29.86% over prior handcrafted and automated workflows, and a heterogeneous population built from four open-weight LLMs outperforms a much stronger closed model while using roughly one-eighth of its inference cost. If right, EvoFlow shifts agentic AI design from picking one architecture to cultivating a portfolio that can be queried by difficulty and budget.

What carries the argument

The load-bearing object is the workflow population $P^{(t)} = \{G_1, \ldots, G_N\}$, where each individual $G = (O^S, E^a)$ is a directed graph of operator nodes; each operator node is a composite of LLM-invoking nodes (model, prompt, temperature). Three mechanisms carry the argument: (1) tag-based retrieval scores workflows against a query by cosine similarity of lightweight semantic embeddings and picks $K$ parents; (2) LLM-facilitated crossover and three mutation functors—LLM mutation $\mu_l$, prompt mutation $\mu_p$, and operator mutation $\mu_o$—generate offspring; (3) niching-based selection identifies a niche by tag and cost ranking, updates cumulative cost and performance records, and eliminates the individual with the worst indicator-based fitness $F(G) = \sum \exp(I(G, G_\circledcirc)/(\phi I_{\max}))$. The combination keeps the population spread along the Pareto front while letting simple and complex workflows coexist, with the named component being a niching evolutionary algorithm driven by a Pareto-dominance-preserving indicator.

What would settle it

Log every offspring workflow produced during evolution and count how many parse and execute without manual repair when EvoFlow is run from scratch on a fresh benchmark split; if the majority fail to execute, or if replacing the mutation operators with random code edits leaves the final Pareto front unchanged, the reported superiority of EvoFlow would not be reproduced.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that cost and performance of an agentic workflow can be treated as two competing objectives in a multi-objective search, and that maintaining a diverse population of heterogeneous, complexity-adaptive workflows—rather than converging to one homogeneous complex workflow—is both feasible and beneficial. The search space is defined over operator nodes $O_j = (I^o_j, E^o_j)$, composite structures built from LLM-invoking nodes $I_i = (M_i, P_i, \tau_i)$, which explicitly includes the choice of LLM backbone, prompt, and temperature in the optimization. The population is initialized from a repository of known operators; each new query triggers tag-based retrieval of parent workflows, LLM-assisted crossover, and mutation of the LLM, prompts, or operator topology; and niching selection retains individuals that are both non-dominated and locally spread in cost-performance space, using a Pareto-dominance-preserving indicator fitness. The reported results are that EvoFlow beats prior automated workflows by 1.23%–29.86% on six benchmarks in the homogeneous setting, and in the heterogeneous setting a population of four open-weight models reaches 72.90% on a MATH subset and 87.62% pass@1 on MBPP, exceeding o1-preview at 12.4% of its inference cost.

Load-bearing premise

The evolution loop assumes that the LLM asked to perform crossover and mutation reliably emits novel, syntactically valid, executable workflow code in the required JSON format; the paper reports no offspring validity rate, execution failure rate, or repair effort, so if that assumption fails the population cannot improve.

Editorial extensions

If this is right

Agentic workflow automation shifts from single-objective performance maximization to a multi-objective cost-performance search, so a user can pick a workflow from the Pareto front according to a budget.
Heterogeneous pools of weaker open-weight models can match or beat a much stronger proprietary model on math and code benchmarks at a fraction of the inference cost.
Cross-domain training benefits population-based search: EvoFlow improves on MBPP when trained jointly on MATH and MBPP, while single-workflow optimizers such as GPTSwarm, DyLAN, and AFlow degrade.
The system exhibits complexity adaptivity: simple I/O workflows are retained for easy queries while complex multi-turn workflows are reserved for hard queries, so per-query cost tracks query difficulty.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Beyond the paper, a learned complexity-aware router trained on the final Pareto population could replace tag-based retrieval with direct budget-conditioned dispatch, making deployment even cheaper.
The operator-graph representation is compatible with grammar-guided genetic programming; constraining crossover and mutation to a grammar of operator templates could raise offspring validity and reduce reliance on the generating LLM's formatting skill.
Because the cost figures depend on vendor API pricing at the time of the experiments, the specific 12.4% ratio is not a permanent property of the method; the structural claim about a cost-performance Pareto front is the part that should persist.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

EvoFlow is the first agentic-workflow search I've seen that treats cost and performance as a true multi-objective problem and uses niching to keep a diverse Pareto population alive. That's a real step beyond ADAS, AFlow, and AgentSquare, which optimize a single workflow against a single objective. The method description is careful, the operator definitions and prompts are in the appendix, and the ablations and sensitivity analyses are useful. The circularity burden is low: train/test splits are standard and the hyperparameters are hand-chosen rather than fitted to benchmark scores.

The soft spots are mostly about evidence hygiene, and one is load-bearing. The abstract says seven benchmarks; the body tests six. The "12.4% of o1-preview's inference cost" claim is actually 12.4% of overall cost. The MATH gain over AFlow is quoted as 5.91% in the intro and 6.42% in Table 2. The MATH subset is described as 617 problems but Table 5 sums to 605. None of these individually sinks anything, but they make the numbers hard to trust. There are no error bars on any main table, which matters for stochastic LLM-based search.

The bigger issue, and the one I'd push on in review, is that the evolution loop never reports an offspring validity rate or execution-failure rate. The crossover and mutation operators ask an LLM to emit executable JSON workflows. Appendix E.3.2 warns against single quotes because they 'may cause execution errors,' which tells me the authors saw malformed generations. If a large fraction of offspring are unparseable or crash, Algorithm 1's selection degrades to random perturbation plus initial-population selection, and the Pareto-front gains in Table 3 and Figure 4 can't be credited to the evolutionary operators. The operator-mutation ablation would be uninterpretable. This is not cosmetic; it's the mechanism the whole paper hangs on.

That said, the core idea holds up. I don't see a methodological error that voids the contribution, just missing reliability evidence that needs to be supplied. The paper deserves peer review, but it needs a major revision that fixes the inconsistencies, adds variance reporting and code, and reports the validity/execution statistics for generated offspring.

I'd bring it to reading group if you work on LLM agents or automated workflow search. I'd cite it as prior work on multi-objective workflow evolution, but not for the specific numbers.

Referee Report

5 major / 5 minor

Summary. The paper introduces EvoFlow, an evolutionary algorithm that searches a population of agentic workflows (graphs of LLM-invoking operator nodes) under a multi-objective cost-performance criterion. Workflows are initialized from operator templates, tagged by an LLM, retrieved by cosine similarity to incoming queries, recombined via LLM-powered crossover and three mutation operators, and selected via niching with a Pareto-preserving indicator. Experiments on six benchmarks (GSM8K, MATH, MultiArith, HumanEval, MBPP, ALFWorld) are reported, along with heterogeneous-model and cross-domain settings, cost analysis, ablations, and sensitivity analyses. The central claim is that EvoFlow produces a diverse Pareto set of workflows that outperforms manual and automated baselines while using much cheaper open-source models.

Significance. If substantiated, EvoFlow would be a useful step beyond single-workflow automation: it frames workflow search as multi-objective, explicitly models LLM heterogeneity, and demonstrates a simple niching-based selection scheme. The paper includes a detailed algorithm description, an operator repository, prompt templates, and ablation studies, and promises code release. However, the evidence as presented has important reliability gaps—notably the absence of offspring-validity statistics and error bars—so the significance claim is presently conditional.

major comments (5)

[Section 4.2, Algorithm 1, Appendix E.3.2] The central claim that evolutionary search improves the population depends on LLM-generated crossover and mutation outputs being syntactically valid and executable, but no offspring validity rate, runtime failure rate, or repair/retry count is reported anywhere. Algorithm 1 filters offspring solely by Pareto dominance-based fitness, which cannot detect structurally invalid workflows; Appendix E.3.2 even warns against single quotes 'as they may cause execution errors,' indicating execution failures occurred during development. Without a validity statistic, the reported gains in Tables 2-3 and Figure 4 cannot be attributed to the evolutionary operators as written, and the ablation in Figure 5 loses interpretability.
[Tables 2 and 3, Figures 5 and 6] All main performance comparisons are reported as single accuracy numbers with no standard deviations, confidence intervals, or number of independent runs. Several differences are small relative to typical benchmark noise (e.g., 0.11-2.15 percentage points on GSM8K, ALFWorld, and MBPP in Table 2), so the claimed improvements over strong baselines such as AgentSquare and AFlow are not statistically established. The authors should add multiple-seed results or at least variance estimates for the main tables and ablations.
[Abstract, Section 1, Table 3] The abstract's headline economy claim is inaccurate. Table 3 shows that 12.4% is the ratio of overall cost (EvoFlow 972.58 vs o1-preview 7840.51 in 10^-3 $), while the inference cost ratio is 513.34/7840.51 = 6.5%. Additionally, the claim that EvoFlow 'surpasses' o1-preview holds on MATH (72.90 vs 70.20) but not on MBPP (87.62 vs 89.65), so the abstract should qualify the claim to MATH and correct the cost figure.
[Abstract, Section 1, Table 2, Section 5.2] The reported gain over AFlow on MATH is inconsistent: Section 1 says 5.91%, while Table 2 and Section 5.2 report 6.42% (57.70 - 51.28). Also, the abstract and contribution list say 'seven benchmarks' while the body explicitly states six benchmarks (Section 5.1) and Table 2 contains six datasets. These numeric inconsistencies should be reconciled.
[Section 5.2, Table 6] The cross-domain claim is one-sided. The text says EvoFlow 'successfully benefits from cross-domain training on MBPP, improving from 87.62% to 88.35%,' but Table 6 shows that the same cross-domain training decreases MATH accuracy from 72.90 to 72.69. The claimed benefit is therefore mixed, and the discussion should acknowledge the trade-off.

minor comments (5)

[Table 2 caption] The caption says 'get-4o-mini'; this should be 'gpt-4o-mini'.
[Figures 1 and 4] The text labels the curve 'Pareto Font'; this should be 'Pareto Front'.
[Section 1, Contributions] The word 'Emperical' should be 'Empirical'.
[Equation (17)] The fitness function is not written consistently: F(G) is defined as a sum over all G in the set, so the right-hand side does not depend on the free variable G. The intended indicator-based sum should use a distinct index, e.g., F(G_i) = sum_{G_j in S, j != i} exp(I(G_i, G_j)/(phi*I_max)).
[Section 5.1, Table 5] The dataset statistics in Table 5 list ALFWorld with 230 train and 327 test examples, but Section G.1 says the train:test split follows a 1:4 ratio except for ALFWorld, which should be clarified.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: EvoFlow's evolution loop is evaluated on held-out tasks with fixed hyperparameters; self-citations are background, not load-bearing.

full rationale

EvoFlow's derivation chain is self-contained with respect to the circularity checks. The method optimizes a population of workflows using training-set queries (Equations 4–6, Algorithm 1), evaluates candidates in the multi-objective space (Equations 15–17), and reports test-set performance on held-out splits (Tables 2, 3, and 6). No parameter is fitted to the test benchmarks: K=3, κ=5, N=15, E=5, and φ=0.05 are fixed constants stated in Section 5.1. The operator repository (Appendix D) is assembled from published structures such as CoT, Debate, and Reflexion, and the LLM-powered crossover and mutation operators (Equations 11–14) do not assume or encode the reported gains; they generate candidates that are then selected by measured performance and cost. The cited prior work by the same authors (e.g., Cut the Crap and G-Designer) appears only as background in Sections 1 and 2 and is not load-bearing for EvoFlow's evolution loop. The absence of offspring validity or execution-failure statistics (Section 4.2 and the Appendix E.3.2 warning about single quotes) is a substantive robustness and correctness concern, but it is not circularity: even if many offspring were invalid, the reported comparison would be threatened by a broken optimization loop, not by a prediction that reduces to its fitted inputs by construction.

Assumptions & free parameters 7 free parameters · 5 assumptions · 0 invented entities

No new physical or conceptual entities are introduced; 'invoking nodes' and 'operator nodes' are compositional abstractions of existing LLM calls. The free parameters are hyperparameters of the evolutionary search, not physical constants. The axioms are the load-bearing premises about LLM reliability, embedding retrieval, dataset transfer, and cost accounting.

free parameters (7)

population size N = 15
Chosen after sensitivity analysis (Section 5.4, Figure 6); larger N improves accuracy but raises per-query cost, so N=15 trades cost for performance.
number of parents K = 3
Set in Section 5.1; sensitivity shows both too-small and too-large K degrade performance.
number of utility tags κ = 5
Set in Section 5.1; governs tag granularity for retrieval.
niching area size E = 5
Set in Section 5.1; controls how many similar workflows compete for survival.
indicator scaling factor φ = 0.05
Fixed following Zitzler and Künzli (2004) in Equation (17).
Self-Refine max iterations = 5
Operator definition in Appendix D; affects cost and accuracy of the Self-Refine operator.
LLM-Debate rounds = 2
Operator definition in Appendix D; up to two debate rounds among three debaters.

assumptions (5)

domain assumption LLM-generated offspring workflows are valid and executable
Crossover (Eq. 11) and mutation (Eqs. 12-14) rely on an LLM producing runnable workflow code in the required format; no validity rate is reported.
domain assumption Tag embeddings predict workflow suitability
Equation (10) retrieves parents by cosine similarity between query embeddings and workflow tags using all-MiniLM-L6-v2; if tag similarity does not track actual aptness, parent selection is noisy.
domain assumption Training queries represent test queries
The population is evolved on Dtrain and evaluated on Dtest (Section 5.1); performance transfer depends on distributional similarity.
domain assumption API cost model captures true cost
Cost columns in Table 3 use per-token prices of the four open-source APIs; cost comparisons inherit any pricing inaccuracies.
standard math Pareto-dominance indicator selection maintains diversity
Equation (17) is the standard I-SDEA indicator from Zitzler and Künzli (2004), assumed to preserve spread along the Pareto front.

how reviews work

0 comments

Cite this review

Pith. "Pith review of EvoFlow: Evolving Diverse Agentic Workflows On The Fly." pith.science (2026). https://pith.science/paper/IFMYWX5L

@misc{pith2026250207373,
  author       = {Pith},
  title        = {Pith review of: EvoFlow: Evolving Diverse Agentic Workflows On The Fly},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/IFMYWX5L}},
  note         = {Machine review of arXiv:2502.07373}
}

abstract

The past two years have witnessed the evolution of large language model (LLM)-based multi-agent systems from labor-intensive manual design to partial automation (\textit{e.g.}, prompt engineering, communication topology) and eventually to fully automated design. However, existing agentic automation pipelines often lack LLM heterogeneity and focus on single-objective performance optimization, limiting their potential to combine weaker models for more customized and cost-effective solutions. To address this challenge, we propose EvoFlow, a niching evolutionary algorithm-based framework to automatically search a population of heterogeneous and complexity-adaptive agentic workflows, rather than a single homogeneous, complex workflow. Technically, EvoFlow performs \textit{(1) tag-based retrieval} to extract parent workflows from an agentic population, evolves new workflows through \textit{(2) crossover} and \textit{(3) mutation}, and employs \textit{(4) niching-based selection} to maintain population diversity and quality. Extensive evaluations across seven benchmarks demonstrate that EvoFlow is: \textbf{(I) diverse}, evolving a population of workflows ranging from simple I/O tasks to complex multi-turn interactions; \textbf{(II) high-performing}, outperforming previous handcrafted and automated workflows by $1.23\%\sim29.86\%$; \textbf{(III) economical}, surpassing powerful \llmname{o1-preview} at $12.4\%$ of its inference cost using weaker open-source models.

Figures

Figures reproduced from arXiv: 2502.07373 by the authors.

**Figure 1.** Paradigm comparison. Baseline methods seek a “onesize-fits-all” complex homogenoues workflow, while EvoFlow optimizes a Pareto set of diverse, heterogenous workflows. tasks, including question answering (Zhu et al., 2024a), data analysis (Hong et al., 2024; Li et al., 2024), decisionmaking (Song et al., 2023), code generation (Shinn et al., 2023), video gaming (Wang et al., 2023), and autonomous driving (Jin et al… view at source ↗

**Figure 2.** The visualization of notations in EvoFlow. Search Space. The search space of EvoFlow is defined hierarchically, with the basic unit being the (LLM-)invoking node. These are further assembled into (composite) operator nodes, which are then combined to form the complete workflow G, as visualized in [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

**Figure 3.** The overall framework of EvoFlow. The fundamental unit is the invoking nodes, which collectively form the operator node. EvoFlow initializes the population by combining multiple operator nodes into a workflow (individual), followed by tag-based retrieval and crossover & mutation to generate novel offspring workflows. The population is updated via niching-based selection. optimization objective of EvoFlow is multi-ob… view at source ↗

Figures from the paper (3 more)

**Figure 4.** Figure 4: The cost-performance plane of workflows from EvoFlow, DyLAN, and AFlow. MATH MBPP Vanillla EvoFlow w/o tag w/o LLM Mutation w/o Prompt Mutation w/o Operator Mutation 80 70 60 [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]

**Figure 5.** Figure 5: The ablation study of EvoFlow. The number of parents 𝐾 The number of tags 𝜅 Population size 𝑁 [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]

**Figure 6.** Figure 6: The parameter sensitivity analysis of EvoFlow. The unit of cost per query (right) and performance (left) is 10−3 · $ and accuracy (%), respectively. We further visualize the optimized heterogeneous population of EvoFlow in [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 12 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

Global Optimization and Inference-Time Region Grafting for Agentic Workflows
cs.CL 2026-08 conditional novelty 6.0 of 10

A training-free method that locally replaces failing regions of an offline-optimized agent workflow at inference time, improving average benchmark performance over prior workflow-optimization baselines.
Who Broke the System? Failure Localization in LLM-Based Multi-Agent Systems
cs.CR 2026-07 conditional novelty 6.0 of 10

AgentLocate localizes multi-agent LLM failures to a responsible agent and earliest decisive step via judge hypotheses, confidence-weighted multi-evaluator verification, and LoRA refinement.
Towards Self-Evolving Agents: A Human-Inspired Adaptive Exploration-Exploitation Framework for Genetic Network Programming
cs.NE 2026-07 conditional novelty 6.0 of 10

HGNP improves GNP and its variants via adaptive crossover protecting high-in-degree nodes later, early-favoring mutation of judgment-to-judgment links, and cycle elimination, with HGNP-SBGNP best on Tileworld.
AgentDropoutV2: Optimizing Information Flow in Multi-Agent Systems via Test-Time Rectify-or-Reject Pruning
cs.AI 2026-02 conditional novelty 6.0 of 10

AgentDropoutV2 intercepts each agent's output, retrieves known error patterns, iteratively corrects the output with targeted feedback, and prunes outputs that remain erroneous, improving multi-agent accuracy.
G-Memory: Tracing Hierarchical Memory for Multi-Agent Systems
cs.MA 2025-06 conditional novelty 6.0 of 10

G-Memory stores past multi-agent teamwork in a three-tier graph and retrieves it to boost performance on five benchmarks.
EMAS: Stabilizing Multi-Agent System Evolution through Evidence-Guided Revision
cs.AI 2026-08 conditional novelty 5.0 of 10

EMAS evolves a multi-agent system's prompts and topology from its own execution traces, accepting a revision only after it recurs across samples and passes a paired validation check.
Cross-Task Experiential Learning on LLM-based Multi-Agent Collaboration
cs.CL 2025-05 conditional novelty 5.0 of 10

MAEL gives each agent in a multi-agent LLM system an experience pool and retrieves high-reward past steps to guide new task solving.
MermaidFlow: Redefining Agentic Workflow Generation via Safety-Constrained Evolutionary Programming
cs.LG 2025-05 conditional novelty 5.0 of 10

Encoding LLM agent workflows as statically checked Mermaid graphs and evolving them with constrained mutations improves solve rates on GSM8K, MATH, HumanEval, and MBPP.
CAFES: A Collaborative Multi-Agent Framework for Multi-Granular Multimodal Essay Scoring
cs.CL 2025-05 conditional novelty 5.0 of 10

A student-teacher multi-agent pipeline with positive-only feedback improves QWK agreement with human essay scores by 21% on a multimodal benchmark, with gains concentrated in traits where baselines were weakest.
A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deployment
cs.CR 2025-04 conditional novelty 5.0 of 10

A large collaborative survey organizes LLM and LLM-agent safety issues into a full-stack lifecycle framework from data preparation to deployment.
An Agentic AI for a New Paradigm in Business Process Development
cs.AI 2025-07 reject novelty 4.0 of 10

Business processes can be modeled as goal-driven agent teams where goals, objects, and agents replace fixed task sequences, and workflows emerge from trigger objects.
Graph-Augmented Large Language Model Agents: Current Progress and Future Prospects
cs.AI 2025-07 conditional novelty 3.0 of 10

A survey that categorizes Graph-augmented LLM Agent research into planning, memory, tool management, and multi-agent design, and outlines open directions.

Reference graph

Works this paper leans on

37 extracted references · 29 canonical work pages · cited by 12 Pith papers

[1]

CoT (Wei et al., 2022) reasoning encourages the LLM to think step by step rather than directly outputting an answer

Chain-of-Thought (CoT). CoT (Wei et al., 2022) reasoning encourages the LLM to think step by step rather than directly outputting an answer. This approach enhances its capability to solve complex problems through intermediate reasoning steps, improving task handling and providing greater transparency in the decision-making process

work page 2022
[2]

LLM-Debate (Du et al., 2023) allows multiple LLMs to debate, leveraging diverse perspectives to identify better solutions

LLM-Debate. LLM-Debate (Du et al., 2023) allows multiple LLMs to debate, leveraging diverse perspectives to identify better solutions. In practice, we initialize three debaters and permit up to two debate rounds

work page 2023
[3]

As proposed by Zheng et al

Take a Step Back. As proposed by Zheng et al. (2023b), this operator prompts the LLM to first consider the principles underlying the task. By focusing on foundational principles, the model enhances its reasoning and delivers more accurate solutions

work page 2023
[4]

Adopting the methodology from Wang et al

Self-Consistency. Adopting the methodology from Wang et al. (2023a), this operator aggregates five CoT reasoning paths and determines the final answer through majority voting

work page 2023
[5]

Following Madaan et al

Self-Refine. Following Madaan et al. (2023), this operator initially generates an answer using CoT reasoning, then prompts the agent to self-reflect iteratively. We set a maximum of five refinement iterations

work page 2023
[6]

Inspired by LLM-Blender (Jiang et al., 2023), this operator involves three LLM-powered agents from different sources outputting answers to the same query

Ensemble. Inspired by LLM-Blender (Jiang et al., 2023), this operator involves three LLM-powered agents from different sources outputting answers to the same query. The pairwise ranking is used to evaluate and aggregate their responses into a final solution

work page 2023
[7]

ReAct. Following (Yao et al., 2023b), this operator enables the agent to leverage versatile tools, including code interpreter, web searching, external knowledge database, etc., to handle diverse user demands

work page
[8]

ExpertPrompt. Similar to AutoGPT (Richards & et al., 2023) and expert prompting (Xu et al., 2023), this operator employs dynamic control flows to allow the agent to decide which expert should be utilized for the task. We respectfully note that the selection of these operators is highly customizable, allowing users the flexibility to incorporate their desi...

work page 2023

Show all 37 references

[9]

We follow the original implementation and settings described in (Zhuge et al., 2024)

GPTSwarm. We follow the original implementation and settings described in (Zhuge et al., 2024)

2024
[10]

The setup adheres to the original settings from (Chen et al., 2023b)

AutoAgents. The setup adheres to the original settings from (Chen et al., 2023b)
[11]

**Example 1 ** - **Code:** {MATH} - **Tags:** Right Triangle, Intermediate Combinatorics, Intermediate Computational Mathematics, Intermediate Permutations, Intermediate Geometry
[12]

- Tags should reflect the primary academic disciplines or difficulty level associated with the MATH dataset

**Example 2 ** - **Code:** {CUSTOM} - **Tags:** Number Theory, Integer Properties, Relatively Prime, Prime Factors, Simple Mathematical Problems **Output Format ** - Provide **5 tags **, separated by commas. - Tags should reflect the primary academic disciplines or difficulty ...
[13]

- **Problem Difficulty: ** Assess the complexity level of the problems (e.g., Beginner, Intermediate, Advanced)

**Focus Areas: ** - **Academic Disciplines: ** Identify the main fields related to the MATH dataset (e.g., Linear Algebra, Mathematics, Calculus). - **Problem Difficulty: ** Assess the complexity level of the problems (e.g., Beginner, Intermediate, Advanced)
[14]

- Ensure the output is a single line containing exactly five tags

**Formatting:** - Do not include any additional text or explanations. - Ensure the output is a single line containing exactly five tags. **Important Notes ** - **Avoid General Tags: ** Do not use overly broad tags such as Artificial Intelligence, Natural Language Processing, o...
[15]

"" E.2. Offspring Generation Prompt Offspring generation PROMPT =

Artificial Intelligence, Natural Language Processing, Reasoning Systems, Advanced Problem Solving, Cognitive Science, Multi-Agent System, AI-enhanced Problem Solving, Problem Solving - *Issue:* These tags are too general and do not focus on the difficulty level or specific aca...
[16]

"" 17 EvoFlow: Evolving Diverse Agentic Workflows On The Fly E.3.2. P ROMPT MUTATION Prompt Mutation

**Large Language Model Mutation ** You can replace the LLM backbone that initializes the operators. Your options are limited to the following 4 choices: - meta-llama/llama-3.1-70b-instruct - qwen/qwen-2.5-72b-instruct - deepseek/deepseek-chat-v2.5 - nousresearch/hermes-3-llama...
[17]

Prompt mutation can enhance the clarity of the agent’s output

**Prompt Mutation ** You can modify the prompts used by invoking nodes, such as incorporating few-shot examples or clarifying task instructions. Prompt mutation can enhance the clarity of the agent’s output. You can also create specific prompts to guide the operator in generat...
[18]

A clear restatement of the problem
[19]

An explanation of the mathematical concepts and theorems involved
[20]

A detailed, logical progression of steps leading to the solution
[21]

Clear explanations for each step, including the reasoning behind it
[22]

All mathematical expressions and equations in LaTeX format
[23]

Visual aids or diagrams if applicable (described in text)
[24]

xxx:{{xxx}}, xxx:{{xxx}}

Make sure the final answer displayed in a boxed LaTeX format." response = await self.custom(input=task, instruction=INSTRUCTION_PROMPT) ’’’ ‘‘‘ - You can also concatenate previously generated string results in the input to provide more comprehensive contextual information: ‘‘‘...

2024
[25]

CoT encourages LLM agents to reason step by step rather than directly producing an answer

CoT. CoT encourages LLM agents to reason step by step rather than directly producing an answer. We adopt the implementation from (Zhang et al., 2022)

2022
[26]

The implementation is based on the code from https://github.com/FranxYao/ 19 EvoFlow: Evolving Diverse Agentic Workflows On The Fly Table 5

ComplexCoT. The implementation is based on the code from https://github.com/FranxYao/ 19 EvoFlow: Evolving Diverse Agentic Workflows On The Fly Table 5. Dataset Statistics. Domain Dataset #Train #Test Metric Code Generation HumanEval 33 131 pass@1 MBPP 86 341 pass@1 Math Reaso...
[27]

Self-consistency. We ensemble five CoT-generated solutions and adopt the implementation from https: //github.com/geekan/MetaGPT/blob/4954729e7564c806d7e58b3ed8b00ef991f889cc/ metagpt/ext/aflow/scripts/operator.py#L93
[28]

We utilize five instances of the same LLM, assigning them distinct roles

LLM-Debate. We utilize five instances of the same LLM, assigning them distinct roles. These agents engage in up to two debate rounds, with the final answer determined via majority voting. Implementation follows https: //github.com/ucl-dark/llm_debate
[29]

The LLM-Blender is powered by two gpt-4o-mini, one Qwen-2.5-72b, and one llama-3.1-70b

LLM-Blender. The LLM-Blender is powered by two gpt-4o-mini, one Qwen-2.5-72b, and one llama-3.1-70b
[30]

We directly adopt the implementation from (Liu et al., 2023)

DyLAN. We directly adopt the implementation from (Liu et al., 2023)

2023
[31]

The implementation is adopted from (Chen et al., 2023d)

AgentVerse. The implementation is adopted from (Chen et al., 2023d)
[32]

For MacNet (Qian et al., 2024), we select the ”MacNet-MESH” variant, which is essentially a densely connected complete graph

MacNet. For MacNet (Qian et al., 2024), we select the ”MacNet-MESH” variant, which is essentially a densely connected complete graph

2024
[35]

Implementation details are directly adopted from (Hu et al., 2024b)

ADAS. Implementation details are directly adopted from (Hu et al., 2024b)
[36]

We employ the modular search framework from (Shang et al., 2024)

AgentSquare. We employ the modular search framework from (Shang et al., 2024). The base LLM is consistently set to gpt-4o-mini, with early stopping patience fixed at 5

2024
[37]

MBPP” represent individual training datasets, while “MATH+MBPP

AFlow. In (Zhang et al., 2024c), AFlow utilizes both gpt-4o-mini and the advanced claude-3.5-sonnet. To ensure fairness in homogeneous settings, we limit AFlow to gpt-4o-mini and set MAX ITERATION =20. H. Supplementary Results 20 EvoFlow: Evolving Diverse Agentic Workflows On ...
[2021]

URL https://openreview.net/forum? id=0IOX0YcCdTn. Song, C. H., Wu, J., Washington, C., Sadler, B. M., Chao, W.-L., and Su, Y . Llm-planner: Few-shot grounded plan- ning for embodied agents with large language models. In Proceedings of the IEEE/CVF International Conference on C...

2023 arXiv
[2023]

Jin, Y ., Shen, X., Peng, H., Liu, X., Qin, J., Li, J., Xie, J., Gao, P., Zhou, G., and Gong, J

Association for Computational Linguistics. Jin, Y ., Shen, X., Peng, H., Liu, X., Qin, J., Li, J., Xie, J., Gao, P., Zhou, G., and Gong, J. Surrealdriver: Designing generative driver agent simulation framework in urban contexts based on large language model, 2023. Khattab, O.,...

2023 arXiv

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.

[1] [1]

CoT (Wei et al., 2022) reasoning encourages the LLM to think step by step rather than directly outputting an answer

Chain-of-Thought (CoT). CoT (Wei et al., 2022) reasoning encourages the LLM to think step by step rather than directly outputting an answer. This approach enhances its capability to solve complex problems through intermediate reasoning steps, improving task handling and providing greater transparency in the decision-making process

work page 2022

[2] [2]

LLM-Debate (Du et al., 2023) allows multiple LLMs to debate, leveraging diverse perspectives to identify better solutions

LLM-Debate. LLM-Debate (Du et al., 2023) allows multiple LLMs to debate, leveraging diverse perspectives to identify better solutions. In practice, we initialize three debaters and permit up to two debate rounds

work page 2023

[3] [3]

As proposed by Zheng et al

Take a Step Back. As proposed by Zheng et al. (2023b), this operator prompts the LLM to first consider the principles underlying the task. By focusing on foundational principles, the model enhances its reasoning and delivers more accurate solutions

work page 2023

[4] [4]

Adopting the methodology from Wang et al

Self-Consistency. Adopting the methodology from Wang et al. (2023a), this operator aggregates five CoT reasoning paths and determines the final answer through majority voting

work page 2023

[5] [5]

Following Madaan et al

Self-Refine. Following Madaan et al. (2023), this operator initially generates an answer using CoT reasoning, then prompts the agent to self-reflect iteratively. We set a maximum of five refinement iterations

work page 2023

[6] [6]

Inspired by LLM-Blender (Jiang et al., 2023), this operator involves three LLM-powered agents from different sources outputting answers to the same query

Ensemble. Inspired by LLM-Blender (Jiang et al., 2023), this operator involves three LLM-powered agents from different sources outputting answers to the same query. The pairwise ranking is used to evaluate and aggregate their responses into a final solution

work page 2023

[7] [7]

ReAct. Following (Yao et al., 2023b), this operator enables the agent to leverage versatile tools, including code interpreter, web searching, external knowledge database, etc., to handle diverse user demands

work page

[8] [8]

ExpertPrompt. Similar to AutoGPT (Richards & et al., 2023) and expert prompting (Xu et al., 2023), this operator employs dynamic control flows to allow the agent to decide which expert should be utilized for the task. We respectfully note that the selection of these operators is highly customizable, allowing users the flexibility to incorporate their desi...

work page 2023