Pith. sign in

REVIEW 4 major objections 5 minor 34 references

Adaptive Multi-Agent Reasoning via Automated Workflow Generation

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper argues that automatically generating a multi-agent workflow—complete with roles, tools, and iterative prompt refinement—can turn an off-the-shelf non-reasoning model into a better solver of novel logic riddles than…

desk verdict A real systems contribution with a suggestive but not yet supported headline claim: the evaluation mixes IPR tuning examples into the final score and omits the raw GPT-4.1 baseline. read the letter →

arxiv 2507.14393 v1 pith:XSBXDSVY submitted 2025-07-18 cs.AI

Humza Sami , Mubashir ul Islam , Pierre-Emmanuel Gaillardon , Valerio Tenace This is my paper
classification cs.AI
keywords multi-agentsystemsautomatedworkflowsynthesisiterativepromptrefinementlargereasoningmodelslogicalriddlebenchmarkgeneralizationGPT-4.1
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Nexus Architect is an automated system that tries to establish that an ordinary, non-reasoning language model can outperform dedicated reasoning models on hard logic puzzles if it is embedded in a self-generated multi-agent workflow. Given a task description and ten example question–answer pairs, the system decomposes the task, designs a team of supervisor and worker agents with tools, validates the workflow on the examples, and then iteratively rewrites each agent's system prompt until failures are corrected. On ArcBench, a curated set of 158 translated and revised riddles, the resulting workflows running GPT-4.1 achieve a mean pass rate of 62.74% and a best of 74.68%, against 44.94% for the strongest tested reasoning model, Gemini 2.5 Flash Preview; the paper reports up to a 66% relative improvement over that model and $2.5\times$ or better over several others. The importance, if the result is right, is that test-time orchestration and automated prompt engineering can substitute for expensive reasoning-specific training.

What carries the argument

The machinery that carries the argument is the automated workflow synthesis pipeline closed by the Iterative Prompt Refinement (IPR) loop. The pipeline starts from a user prompt and example pairs, decomposes the task into requirements, designs a multi-agent blueprint, instantiates supervisors, workers, and tools through component builders, and validates the running workflow on the supplied examples. When validation fails, failure cases are analyzed and fed back to a Prompt Engineering stage, which rewrites the agents' system prompts for the next iteration; this reinforcement-style loop continues until performance criteria are met. IPR is the load-bearing part for the generalization claim because it improves behavior by changing instructions rather than model weights, and the paper reports that pass rates rise across five iterations despite only 10 of 158 questions being used for tuning.

What would settle it

Hold out every question used during IPR (or all ten tuning examples per run) from the final scored set and recompute the pass rate; if the advantage over Gemini 2.5 Flash Preview shrinks to near zero, the reported 62.74% mean is driven by prompt fitting rather than transferable reasoning.

Watch

Extended reading notes

Core claim

The discovery, stated on the paper's own terms, is that an automatically synthesized, validated, and iteratively refined multi-agent workflow makes an off-the-shelf non-reasoning model (GPT-4.1, used without fine-tuning) exceed the pass rate of state-of-the-art LRMs on a benchmark of novel logical riddles. The Architect generates the whole reasoning pipeline from a prompt and ten examples: it decomposes the task, selects strategies, tools, and adversarial techniques, builds the supervisor/worker topology, and then applies the IPR feedback loop, which converts validation failures into targeted modifications of system messages. Across five runs this yields a mean pass rate of 62.74% and a best of 74.68% on ArcBench, versus 44.94% for Gemini 2.5 Flash Preview, about 30.38% for Claude Sonnet 4 and DeepSeek-R1, and 24.68% for Llama 4 Scout. The paper reads this as evidence that robust reasoning can be democratized through workflow design and agentic automation rather than through ever-larger or reasoning-specialized models.

Load-bearing premise

The load-bearing premise is that tuning system prompts on ten examples drawn from the same ArcBench pool that is later scored still measures generalization to unseen problems, rather than fitting the workflow to the test set.

Editorial extensions

If this is right

  • An off-the-shelf non-reasoning model can beat dedicated reasoning models on a logic-riddle benchmark when the workflow around it is generated automatically.
  • Reasoning gains can be obtained at test time through orchestration and prompt refinement, without fine-tuning or training a larger model.
  • Because each run tunes on only 10 of 158 questions yet raises final pass rates, the paper argues the improvement reflects generalization rather than simple memorization of the ArcBench items.
  • Open-source release of Nexus Architect and ArcBench allows the workflow-synthesis and IPR results to be reproduced and applied to other problem classes.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The IPR examples are sampled from the same 158-question pool used to compute final pass rates, so part of the reported advantage may reflect fitting system prompts to the test distribution; a held-out tuning set would separate transferable gains from fitting.
  • The appendix's digital-watch example suggests the IPR gains concentrate on riddle conventions—recognizing trick or punchline answers—so the margin over reasoning models could be smaller on formal, multi-step mathematical reasoning.
  • A direct test of transferability would synthesize a workflow on one family of problems (say arithmetic word problems), then deploy it unchanged on a disjoint family; the paper reports no such cross-family evaluation.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces Nexus Architect, an extension of the Nexus multi-agent framework that automatically synthesizes a multi-agent reasoning workflow from a user prompt and a small set of problem-solution pairs, and then iteratively refines agent system prompts through an Iterative Prompt Refinement (IPR) loop. The authors evaluate the system on ArcBench, a revised English subset of the RoR-Bench riddle benchmark, using GPT-4.1 as the underlying non-reasoning LLM, and compare pass rates against Llama 4 Scout/Maverick, DeepSeek-R1, Claude 3.5 Sonnet/Sonnet 4/Opus 4, and Gemini 2.5 Flash Preview. They report that Architect consistently outperforms all baselines, with a best-case pass rate of 74.68% versus 44.94% for Gemini 2.5 Flash Preview, and interpret this as evidence that automated workflow generation and prompt refinement improve the generalization capabilities of standard LLMs.

Significance. If the result holds on a properly held-out evaluation, the paper would make a useful empirical contribution: it demonstrates an automated alternative to fine-tuning for reasoning tasks, and it ships open-source code and a benchmark. The five independent runs and the detailed configuration table are positive features. However, the central generalization claim is currently not established because the IPR tuning examples are included in the final pass rate, and the contribution of the multi-agent workflow over raw GPT-4.1 is not quantified. The open-source release and the clear description of the workflow synthesis mechanism are strengths that support reproducibility.

major comments (4)
  1. [Section 4.1, Figure 3] The final pass rate is computed over the entire ArcBench dataset, which includes the 10 question-answer pairs used for IPR tuning in each run. Section 4.1 states that 10 pairs are 'instrumental for the IPR phase,' and Figure 3 shows the final pass rate as a solid line over the whole dataset. Since the system prompts are refined on the exact questions and answers that later count toward the reported score, the reported 62.74% mean and 74.68% max are not measurements of generalization to unseen problems. Direct leakage is bounded at about 6.3 percentage points, so it cannot by itself explain the gap over the best baseline, but the paper's central claim that IPR improves generalizability (Section 4.3) requires reporting pass rates on the held-out subset (the 148 items not used in IPR) or using a separate tuning set. Without this, the abstract's phrase 'generalize to novel, unseen problems' is unsupported.
  2. [Section 4.2, Table 1] There is no standalone GPT-4.1 baseline. The comparison in Figure 2 is between Architect, which runs GPT-4.1, and other models, but the paper never reports how GPT-4.1 alone performs on ArcBench with equivalent prompting and decoding settings. As a result, the contribution of the multi-agent workflow, the automated workflow synthesis, and the IPR loop over the raw model cannot be separated. The authors should add a GPT-4.1 zero-shot baseline, and ideally a GPT-4.1 baseline that receives the same 10 examples as in-context exemplars without the IPR loop.
  3. [Section 4.2, Figure 2] The five Architect runs produce pass rates from 51.27% to 74.68%, yet all baseline models are reported as single point estimates without confidence intervals or significance tests. The claim 'consistently outperforms' and the precise multipliers (2.5x, 3x) are therefore only point estimates under unknown sampling variability. The authors should report per-run standard deviations or bootstrapped confidence intervals and, where differences are considered, a paired significance test across the 158 items.
  4. [Section 4.1, ArcBench construction] ArcBench is a revised version of RoR-Bench in which the authors 'revised several questions and answers to enhance their accessibility' and translated the set into English. Because the benchmark is authored and modified by the same group that evaluates its own system, the revision criteria are a free parameter that could bias results in favor of the proposed pipeline. The authors should justify the revisions with explicit criteria, report results on the original RoR-Bench items as a sanity check, and release the mapping between original and revised items.
minor comments (5)
  1. [Figure 2] The labels 'mean=62.74 max=74.68 min=51.27' appear in the plot area without a legend entry explaining that they refer to the five Nexus Architect runs; this should be clarified in the caption.
  2. [Section 4.1] The pass rate metric is defined only as 'the ratio of correct answers over the total number of questions.' The paper should specify how answers are extracted and scored (exact match, LLM judge, or human review), how many samples per question were used, and whether the baselines were evaluated with greedy decoding or sampling.
  3. [Appendix A, Table 1] The configuration table lists temperature and top_p for each model, but the paper does not state whether the reported pass rates are averaged over multiple samples; given that GPT-4.1 is used with temperature and top_p both set to 1, stochasticity should be quantified.
  4. [Section 4.1, Appendix B] The revised benchmark name ArcBench is introduced in Section 4.1, but the introduction (list item 3) says it is 'derived from RoR-Bench'; the appendix or repository should include the full revision log to support reproducibility.
  5. [Appendix B] Appendix B refers to the IPR examples as 'training example' and 'representative training example'; this terminology should be reconciled with the claim that the evaluation measures generalization, since the examples are drawn from the same benchmark.

Circularity Check

1 steps flagged · score 6.0 of 10

IPR tuning examples are included in the reported full-dataset pass rate, so the generalization claim is partially fitted rather than held out.

  1. fitted input called prediction [Section 4.1 (Data Preparation & Methodology) and Section 4.3 (IPR Performance Assessment), Figure 3]
    "For each of five independent runs, we randomly selected 10 question-answer pairs as representative task examples instrumental for the IPR phase (see Figure 1), resulting in five IPR iterations per run. ... The solid line indicates the final pass rate over the entire dataset."

    The IPR loop refines agent system prompts on 10 ArcBench question-answer pairs, and the final pass rate plotted as the solid line in Figure 3 (and the Nexus Architect bars in Figure 2) is computed over the whole 158-item dataset, which includes those same 10 tuning pairs. Therefore the reported 'generalizability' result is not a purely held-out measurement: up to roughly 6.3 percentage points of the final pass rate can reflect direct in-sample improvement on the exact examples used to tune the prompts. Section 4.3 argues that because only 6% of samples were used, the results show the approach 'is highly effective in improving the generalizability,' but this inference conflates tuning-set performance with test performance.

full rationale

The paper's core contribution is an empirical system comparison, not a mathematical derivation, so most circularity patterns do not apply. The self-citation to the authors' Nexus framework [12] is used as the underlying platform and as in-context documentation, but it is not invoked as an unverified uniqueness theorem or as a substitute for evidence, so it is not load-bearing circularity. ArcBench is derived from the external RoR-Bench suite, which provides independent grounding. However, there is one genuine fitted-input-called-prediction issue: the IPR phase tunes system prompts on 10 randomly selected ArcBench pairs, and the final full-dataset pass rate reported in Figures 2 and 3 includes those same pairs. Consequently, part of the reported gain over LRMs is attributable to in-sample tuning rather than to generalization to unseen problems. The direct contamination is bounded at about 6.3 percentage points, so it cannot fully explain the roughly 18-30 point gap over the best baseline, and the majority of the evaluation remains held out. Still, the paper's specific claim in Section 4.3 that using only 6% of samples demonstrates strong generalizability is weakened because those samples are scored in the final metric. The evaluation also omits a standalone GPT-4.1 baseline, which complicates attribution of the gain to the multi-agent workflow, though this is a completeness concern rather than circularity. Overall, the central claim has substantial independent content, but the IPR generalization claim is partially circular by construction.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central claim depends on the correctness of the revised benchmark labels and on an evaluation protocol that separates prompt tuning from held-out testing; both are assumptions rather than established facts.

free parameters (4)
  • IPR iteration count = 5
    Five feedback rounds were used in every run; more or fewer rounds would change the reported pass rates.
  • In-context example set size = 10 of 158 (about 6%)
    Ten randomly sampled pairs are used to fit system prompts; this fraction dictates how much of the final score is tuned.
  • GPT-4.1 sampling temperature and top_p = temperature 1, top_p 1
    Chosen for all Architect runs; no sensitivity analysis is reported.
  • ArcBench revision criteria = unspecified
    The number and nature of question and answer revisions are not quantified, making the benchmark's difficulty a hand-adjusted variable.
assumptions (4)
  • domain assumption ArcBench answer labels are correct after author revision
    Section 4.1: the authors translate and revise RoR-Bench items; no external validation or scoring rubric is provided for the revised labels.
  • domain assumption Pass rate is computed by a well-defined answer-matching procedure
    Section 4.2 defines pass rate as ratio of correct answers, but the matching method (exact string, rubric, or LLM judge) is not described.
  • domain assumption Nexus documentation summary is a sufficient grounding source for workflow synthesis
    Section 3: the workflow generator uses a pre-synthesized summary of the Nexus GitHub repository as its in-context knowledge source.
  • domain assumption Single-run baseline pass rates are representative
    Figure 2 reports one value per baseline model; no repeated trials or confidence intervals are given.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Adaptive Multi-Agent Reasoning via Automated Workflow Generation." pith.science (2026). https://pith.science/paper/XSBXDSVY

@misc{pith2026250714393,
  author       = {Pith},
  title        = {Pith review of: Adaptive Multi-Agent Reasoning via Automated Workflow Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XSBXDSVY}},
  note         = {Machine review of arXiv:2507.14393}
}
abstract

The rise of Large Reasoning Models (LRMs) promises a significant leap forward in language model capabilities, aiming to tackle increasingly sophisticated tasks with unprecedented efficiency and accuracy. However, despite their impressive performance, recent studies have highlighted how current reasoning models frequently fail to generalize to novel, unseen problems, often resorting to memorized solutions rather than genuine inferential reasoning. Such behavior underscores a critical limitation in modern LRMs, i.e., their tendency toward overfitting, which in turn results in poor generalization in problem-solving capabilities. In this paper, we introduce Nexus Architect, an enhanced iteration of our multi-agent system framework, Nexus, equipped with a novel automated workflow synthesis mechanism. Given a user's prompt and a small set of representative examples, the Architect autonomously generates a tailored reasoning workflow by selecting suitable strategies, tool integrations, and adversarial techniques for a specific problem class. Furthermore, the Architect includes an iterative prompt refinement mechanism that fine-tunes agents' system prompts to maximize performance and improve the generalization capabilities of the system. We empirically evaluate Nexus Architect by employing an off-the-shelf, non-reasoning model on a custom dataset of challenging logical questions and compare its performance against state-of-the-art LRMs. Results show that Nexus Architect consistently outperforms existing solutions, achieving up to a 66% increase in pass rate over Gemini 2.5 Flash Preview, nearly 2.5$\times$ against Claude Sonnet 4 and DeepSeek-R1, and over 3$\times$ w.r.t. Llama 4 Scout.

Figures

Figures reproduced from arXiv: 2507.14393 by the authors.

Figure 1
Figure 1. Block diagram of the proposed Nexus Architect. Starting from a user prompt, the system decomposes the task, [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Performance comparison between our five Architect-generated workflows and recently released state-of-the-art [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Pass rates observed across IPR iterations. Bars, grouped by Architect run, show the pass rates achieved at [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

34 extracted references · 20 canonical work pages

  1. [1]

    The illusion of thinking: Understanding the strengths and limitations of reasoning models via the lens of problem complexity,

    P. Shojaee, I. Mirzadeh, K. Alizadeh, M. Horton, S. Bengio, and M. Farajtabar, “The illusion of thinking: Understanding the strengths and limitations of reasoning models via the lens of problem complexity,” 2025. [Online]. Available: https://ml-site.cdn-apple.com/papers/the-illusion-of-thinking.pdf

    work page 2025

  2. [2]

    What has a foundation model found? using inductive bias to probe for world models,

    K. Vafa, P. G. Chang, A. Rambachan, and S. Mullainathan, “What has a foundation model found? using inductive bias to probe for world models,” arXiv preprint arXiv:2507.06952, 2025

    arXiv 2025

  3. [3]

    Faith and fate: Limits of transformers on compositionality,

    N. Dziri, X. Lu, M. Sclar, X. L. Li, L. Jiang, B. Y . Lin, S. Welleck, P. West, C. Bhagavatula, R. Le Braset al., “Faith and fate: Limits of transformers on compositionality,” Advances in Neural Information Processing Systems, vol. 36, pp. 70 293–70 332, 2023

    work page 2023

  4. [4]

    Unveiling causal reasoning in large language models: Reality or mirage?

    H. Chi, H. Li, W. Yang, F. Liu, L. Lan, X. Ren, T. Liu, and B. Han, “Unveiling causal reasoning in large language models: Reality or mirage?” Advances in Neural Information Processing Systems, vol. 37, pp. 96 640–96 670, 2024

    work page 2024

  5. [5]

    Large language models are not strong abstract reasoners,

    G. Gendron, Q. Bao, M. Witbrock, and G. Dobbie, “Large language models are not strong abstract reasoners,” arXiv preprint arXiv:2305.19555, 2023

    arXiv 2023

  6. [6]

    Generalization vs memorization: Tracing language models’ capabilities back to pretraining data,

    X. Wang, A. Antoniades, Y . Elazar, A. Amayuelas, A. Albalak, K. Zhang, and W. Y . Wang, “Generalization vs memorization: Tracing language models’ capabilities back to pretraining data,” arXiv preprint arXiv:2407.14985, 2024

    arXiv 2024

  7. [7]

    On memorization of large language models in logical reasoning,

    C. Xie, Y . Huang, C. Zhang, D. Yu, X. Chen, B. Y . Lin, B. Li, B. Ghazi, and R. Kumar, “On memorization of large language models in logical reasoning,” arXiv preprint arXiv:2410.23123, 2024

    arXiv 2024

  8. [8]

    Recitation over reasoning: How cutting-edge language models can fail on elementary school-level reasoning problems?

    K. Yan, Y . Xu, Z. Du, X. Yao, Z. Wang, X. Guo, and J. Chen, “Recitation over reasoning: How cutting-edge language models can fail on elementary school-level reasoning problems?” arXiv preprint arXiv:2504.00509, 2025

    arXiv 2025

Show all 34 references
  1. [9]

    Reasoning or reciting? exploring the capabilities and limitations of language models through counterfactual tasks,

    Z. Wu, L. Qiu, A. Ross, E. Akyürek, B. Chen, B. Wang, N. Kim, J. Andreas, and Y . Kim, “Reasoning or reciting? exploring the capabilities and limitations of language models through counterfactual tasks,” in Proceedings of the 2024 Conference of the North American Chapter of th...

    2024

  2. [10]

    What do learning dynamics reveal about generalization in llm reasoning?

    K. Kang, A. Setlur, D. Ghosh, J. Steinhardt, C. Tomlin, S. Levine, and A. Kumar, “What do learning dynamics reveal about generalization in llm reasoning?” arXiv preprint arXiv:2411.07681, 2024

    2024 arXiv

  3. [11]

    The reasoning-memorization interplay in language models is mediated by a single direction,

    Y . Hong, D. Zhou, M. Cao, L. Yu, and Z. Jin, “The reasoning-memorization interplay in language models is mediated by a single direction,” arXiv preprint arXiv:2503.23084, 2025

    2025 arXiv

  4. [12]

    Nexus: A lightweight and scalable multi-agent framework for complex tasks automation,

    H. Sami, M. ul Islam, S. Charas, A. Gandhi, P.-E. Gaillardon, V . Tenaceet al., “Nexus: A lightweight and scalable multi-agent framework for complex tasks automation,” arXiv preprint arXiv:2502.19091, 2025

    2025 arXiv

  5. [13]

    Intelligent agents: Theory and practice,

    M. Wooldridge and N. R. Jennings, “Intelligent agents: Theory and practice,” The knowledge engineering review, vol. 10, no. 2, pp. 115–152, 1995

    1995

  6. [14]

    Multiagent systems: A survey from a machine learning perspective,

    P. Stone and M. Veloso, “Multiagent systems: A survey from a machine learning perspective,”Autonomous Robots, vol. 8, pp. 345–383, 2000

    2000

  7. [15]

    Generative agents: Interactive simulacra of human behavior,

    J. S. Park, J. O’Brien, C. J. Cai, M. R. Morris, P. Liang, and M. S. Bernstein, “Generative agents: Interactive simulacra of human behavior,” in Proceedings of the 36th annual acm symposium on user interface software and technology, 2023, pp. 1–22

    2023

  8. [16]

    ReAct: Synergizing reasoning and acting in language models,

    S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. Narasimhan, and Y . Cao, “ReAct: Synergizing reasoning and acting in language models,” arXiv preprint arXiv:2210.03629, 2022

    2022 arXiv

  9. [17]

    (2025) AutoGPT: Build, Deploy, and Run AI Agents

    AutoGPT. (2025) AutoGPT: Build, Deploy, and Run AI Agents. [Online]. Available: https://github.com/ Significant-Gravitas/AutoGPT

    2025

  10. [18]

    Hugginggpt: Solving ai tasks with chatgpt and its friends in hugging face,

    Y . Shen, K. Song, X. Tan, D. Li, W. Lu, and Y . Zhuang, “Hugginggpt: Solving ai tasks with chatgpt and its friends in hugging face,” Advances in Neural Information Processing Systems, vol. 36, 2024

    2024

  11. [19]

    (2025) LangGraph

    LangChain. (2025) LangGraph. [Online]. Available: https://github.com/langchain-ai/langgraph

    2025

  12. [20]

    Autogen: Enabling next-gen llm applications via multi-agent conversation framework,

    Q. Wu, G. Bansal, J. Zhang, Y . Wu, S. Zhang, E. Zhu, B. Li, L. Jiang, X. Zhang, and C. Wang, “Autogen: Enabling next-gen llm applications via multi-agent conversation framework,”arXiv preprint arXiv:2308.08155, 2023

    2023 arXiv

  13. [21]

    (2025) CrewAI: Production-grade framework for orchestrating sophisticated AI agent systems

    CrewAI. (2025) CrewAI: Production-grade framework for orchestrating sophisticated AI agent systems. [Online]. Available: https://github.com/crewAIInc/crewAI 7 Adaptive Multi-Agent Reasoning via Automated Workflow Generation

    2025

  14. [22]

    Model Context Protocol (MCP),

    Anthropic, “Model Context Protocol (MCP),” https://docs.anthropic.com/en/docs/agents-and-tools/mcp, 2025, accessed: 2025-05-31

    2025

  15. [23]

    Prewrite: Prompt rewriting with reinforcement learning,

    W. Kong, S. A. Hombaiah, M. Zhang, Q. Mei, and M. Bendersky, “Prewrite: Prompt rewriting with reinforcement learning,” arXiv preprint arXiv:2401.08189, 2024

    2024 arXiv

  16. [24]

    System prompt optimization with meta-learning,

    Y . Choi, J. Baek, and S. J. Hwang, “System prompt optimization with meta-learning,” arXiv preprint arXiv:2505.09666, 2025

    2025

  17. [25]

    Llama 4 scout and maverick: Mixture-of-experts multimodal models,

    M. L. Team, “Llama 4 scout and maverick: Mixture-of-experts multimodal models,” https://github.com/ meta-llama/llama-models/blob/main/models/llama4/MODEL_CARD.md, 2025

    2025

  18. [26]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning,

    D. Guo, D. Yang, H. Zhang, J. Song, R. Zhang, R. Xu, Q. Zhu, S. Ma, P. Wang, X. Bi et al., “Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning,” arXiv preprint arXiv:2501.12948, 2025

    2025 arXiv

  19. [27]

    Claude 3.5 Sonnet,

    Anthropic, “Claude 3.5 Sonnet,” https://www.anthropic.com/news/claude-3-5-sonnet, 2025, accessed: 2025-07- 15

    2025

  20. [28]

    Introducing Claude 4,

    ——, “Introducing Claude 4,” https://www.anthropic.com/news/claude-4, 2025, accessed: 2025-05-31

    2025

  21. [29]

    Gemini 2.5 flash preview – model card,

    Google AI, “Gemini 2.5 flash preview – model card,” PDF document, Jun. 2025. [Online]. Available: https://storage.googleapis.com/model-cards/documents/gemini-2.5-flash-preview.pdf 8 Adaptive Multi-Agent Reasoning via Automated Workflow Generation Appendix A Model Configuration...

    2025

  22. [30]

    Initial System Message Design: The process begins with a carefully crafted system message for the Supervisor agent, responsible for coordinating the other agents

  23. [31]

    Performance Testing: The agent’s performance is evaluated using problem-answer example pairs

  24. [32]

    This feedback identifies issues, root causes, and required changes, and is sent to the Prompt Engineering stage

    Feedback Generation: If the agent fails to produce the expected output, detailed feedback is generated. This feedback identifies issues, root causes, and required changes, and is sent to the Prompt Engineering stage

  25. [33]

    System Message Refinement: Based on the feedback, the system message is modified to address the identified shortcomings

  26. [34]

    meta" answer that prevents classic/technical solution. In several cases, Supervisor did not explicitly require agents to surface

    Verification: The refined system message is tested against the same example to confirm improvement. In the following sections, we walk through a full iteration of this process using a sample riddle involving a digital watch. Initially, the agent’s system message failed to reco...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.