Pith. sign in

REVIEW 4 major objections 4 minor 1 cited by

FlowForge: Guiding the Creation of Multi-agent Workflows with Design Space Visualization as a Thinking Scaffold

T0 review · 4 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read FlowForge claims that making the multi-agent workflow design space visible at three abstraction levels, with in-situ design-pattern suggestions, lets practitioners build working workflows faster and explore more alternatives than a…

desk verdict A well-designed tool with a confounded user study: the automatic GPT-4o generation, not the visualization scaffold, likely drives the reported efficiency and diversity gains. read the letter →

arxiv 2507.15559 v1 pith:FP4WTP5F submitted 2025-07-21 cs.HC

Pan Hao , Dongyeop Kang , Nicholas Hinds , Qianwen Wang This is my paper
classification cs.HC
keywords multi-agentworkflowsLLMdesignspacevisualizationpatternshierarchicalabstractionvisualanalyticsworkflowauthoringin-situguidance
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that the bottleneck in building multi-agent LLM workflows is not implementation support but the exploratory design process, where practitioners commit to one structure early and then tweak prompts, missing better architectures. FlowForge addresses this by visualizing the workflow design space as three hierarchical levels—task planning, agent assignment, and agent optimization—so users can compare alternatives at each level before committing. In a user study against a standard visual programming interface, participants using FlowForge produced their first runnable workflow in about 9.9 minutes versus 18 minutes, created more workflows, and used a wider variety of agent communication patterns. If the claim holds, design-space visualization can serve as a thinking scaffold that prevents design fixation and makes multi-agent workflow creation faster and more exploratory.

What carries the argument

The load-bearing mechanism is the three-level abstraction of the multi-agent workflow design space: task planning (decomposing the task into subtasks), agent assignment (choosing cooperation patterns such as sequential, parallel, reflection, redundant, supervision, or discussion), and agent optimization (configuring individual agents with prompts, tools, and data). FlowForge represents these levels in a hierarchical tree and a scatter plot whose axes are performance dimensions, using custom glyphs—arcs, bars, and file icons—to encode subtask count, parallelism, and computational cost at each level. A canvas view with semantic zooming lets users inspect and edit a workflow, while design-pattern cards offer in-situ recommendations. This hierarchy is what carries the argument: it is both the visual scaffold and the source of the step-by-step guidance that the paper says prevents design fixation and unstructured trial-and-error.

What would settle it

Run the same comparative study with a control version that offers the same generated candidate workflows and design-pattern cards but no hierarchy and no scatter plot; if users explore just as many architectures and take just as long, the three-level scaffold is not what carries the effect. A second check is to give practitioners a task that requires dynamic or human-in-the-loop execution and see whether FlowForge's design space blocks or misleads them.

Watch

Extended reading notes

Core claim

The central discovery is that embedding an explicit, hierarchical visualization of the design space directly into a workflow authoring tool changes how practitioners design: instead of jumping straight to a concrete implementation and iterating on prompts, users move from abstract task decomposition to agent assignment to agent optimization, comparing glyph-encoded alternatives along the way. The tool generates candidate workflows, shows them in a coordinated tree and scatter plot, and surfaces context-aware design-pattern cards matched to the current subtask. In the comparative study with nine participants, FlowForge yielded faster first workflows, more parallel subtasks, and communication patterns beyond single-agent chains, which the paper attributes to the support for divergence-before-convergence and to in-situ guidance. The paper frames this as evidence that design space visualization works as a thinking scaffold for both humans and language models in workflow development.

Load-bearing premise

The paper assumes that the three-level abstraction derived from four interviews and a review of 43 multi-agent systems is a true and useful map of the design space; if the taxonomy misses important architectures—and the paper concedes it excludes dynamic execution and human-in-the-loop changes—the guided exploration could narrow users' options instead of broadening them.

Editorial extensions

If this is right

  • Practitioners can reach a first runnable multi-agent workflow in roughly half the time, with average first-workflow completion dropping from 18.00 to 9.89 minutes.
  • Users explore a wider range of designs: participants created more parallel subtasks and adopted reflection, supervision, redundancy, and discussion patterns, while baseline workflows used only single agents.
  • Design-pattern suggestions get used at the moment of decision, helping users fix performance problems at the architectural level rather than by prompt tweaks alone.
  • Workflow alternatives can be compared across latency, cost, and custom quality ratings before a final structure is committed.
  • The three-level scaffold can act as a thinking scaffold for LLM system design generally, not only as an editing interface.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A likely extension is that similar hierarchical design-space views could support other LLM artifacts, such as prompt suites or evaluation pipelines, where premature convergence is also common; the paper does not test this.
  • The measured gains combine the hierarchy, the scatter plot, and the pattern cards, so a follow-up that separates these components would show which feature actually drives faster and more diverse design.
  • Because the candidate workflows and recommendations are generated by an LLM, the quality ceiling of the scaffold is partly set by the generator; improved generation should compound the benefits.
  • The design space explicitly excludes dynamic and human-in-the-loop architectures, so users who need those may find the scaffold narrowing; testing with such tasks would reveal whether a fourth level or a dynamic dimension is needed.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. FLOWFORGE is an interactive visualization tool that structures the multi-agent workflow design space into three hierarchical levels (task planning, agent assignment, agent optimization), visualizes candidate workflows through a coordinated tree view and scatter plot with novel glyphs, and provides in-situ design-pattern cards. The paper reports a formative interview study, a literature-derived pattern table, two case studies, and a within-subjects user study (n=9) comparing FLOWFORGE against LangGraph Studio. The central claim is that design-space visualization and in-situ guidance improve workflow creation efficiency, exploration diversity, and output quality relative to a baseline system.

Significance. If the central causal claim were established, the paper would make a useful contribution to visualization for LLM-based workflow design: it proposes a concrete three-level abstraction, a coordinated glyph-based interface, and a pattern-recommendation mechanism, and it ships an open-source implementation and demo. The two case studies show plausible real-world utility, and the limitations section is candid about unsupported dynamic architectures and simple generation methods. However, the evaluation as reported does not isolate the visualization scaffold from the embedded GPT-4o generation, and the output-quality component of the central claim is never measured. These gaps are load-bearing for the paper's main conclusion, so the current evidence supports a system-level proof of concept more strongly than it supports the causal 'thinking scaffold' claim.

major comments (4)
  1. [§4.5, §6.1–6.2] The main independent variable in the user study is not the visualization alone. In the FLOWFORGE condition, GPT-4o automatically generates Level-1 task decompositions, Level-2 pattern recommendations, and Level-3 agent configurations (§4.5), while in the baseline condition participants manually construct workflows from scratch (§6.1). The observed 8-minute reduction in first-workflow time (9.89 vs 18.0 minutes) and the higher count of communication patterns (35 instances vs 22 single-agent instances, §6.2) can therefore be explained by the LLM-generated starting points and pattern suggestions rather than by the hierarchical tree, glyphs, scatter plot, or semantic zoom. To support the paper's causal claim, the design should separate automatic generation from the visualization scaffold, for example by comparing FLOWFORGE against a version with generated candidates but no design-space views, or by analyzing how much of each final workflow is user-edited versus generated.
  2. [§6.2, Introduction] One of the three stated benefits in the Introduction—producing higher-quality outputs—is never measured. Section 6.2 reports usability ratings, time-to-first-workflow, and diversity counts, but no rubric, expert rating, or task-specific quality metric for the generated scripts or workflows. Without a quality outcome, the conclusion that FLOWFORGE produces higher-quality outputs is unsupported. This is a load-bearing gap and should be addressed with an explicit quality evaluation, preferably with independent raters blind to condition.
  3. [§6.2] The efficiency result is reported as t=-8.1, p=.003 with n=9. For a paired design, this test statistic and p-value are not mutually consistent, and the paper does not state the test used, the effect size, or whether a correction for multiple comparisons was applied. Moreover, the 25-minute imputation for participants who did not finish in the baseline censors the distribution; a sensitivity analysis (e.g., reporting medians and non-parametric tests, or analyzing time-to-first-run only among participants who finished) would make the efficiency claim more robust.
  4. [§3.2.3, §7.2] The three-level abstraction is the structural backbone of the tool, but its derivation from four interviews and axial coding of 43 systems is not validated with any inter-rater reliability, external agreement, or member-checking. The paper itself notes in §7.2 that this abstraction is not the only viable scaffold; that is an honest limitation, but it also means the current study cannot show whether the taxonomy broadens or inadvertently narrows exploration. A lightweight external validation (e.g., asking independent researchers to map their own workflows onto the three levels, or systematically comparing with the decision tree in [57]) would increase confidence in the design-space claim.
minor comments (4)
  1. [§4.1] There is a typo: 'breath-first exploration' should be 'breadth-first exploration'.
  2. [Figure 9] Figure 9 contains unexplained numeric rows (e.g., '1.777777778', '-0.222222222') that appear to be data artifacts; these should be removed or the underlying values should be presented in a clearly labeled table.
  3. [Figure 10] The caption and panel labels (A/B) in Figure 10 are somewhat confusing; the right panel shows per-participant workflow counts and time allocations, but this is not made explicit in the caption.
  4. [References] Several in-text citations use inconsistent formatting, such as 'Liu et al..' and 'Grunde-McLaughlin et al..' with a double period; these should be corrected for consistency.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the central effectiveness claims are evaluated against an external baseline, and the three-level design abstraction is a grounded design synthesis rather than a result predicted from the tool's own definitions.

full rationale

FLOWFORGE's main claims are tested in Section 6 against LangGraph Studio, an external baseline, using measured completion times, self-reported usability ratings, and manually coded workflow analyses; there are no fitted parameters, equations, or internally derived predictions whose inputs coincide with the claimed outputs. The three-level abstraction in Section 3.2.3 is explicitly grounded in a formative study and a literature review via axial coding, and Section 7.2 acknowledges that 'the three-level abstraction of design space is not the only viable scaffold,' so the taxonomy is not presented as a forced or predicted result. The self-citation in Section 3.2.3 (citation [48], alongside the independent Munzner nested-model reference [34]) is background motivation, not load-bearing evidence, and the disclosed author-participant in Section 3.1 ('One expert is the author of this paper') is a methodological transparency note rather than a circular derivation. The more substantive issues are evidentiary, not circular: FLOWFORGE bundles GPT-4o-based workflow and suggestion generation (Section 4.5) with the visualization, so the efficiency and diversity gains in Section 6.2 cannot be cleanly attributed to the visualization scaffold alone, and the claimed 'higher-quality outputs' are never operationalized with a reported metric. These are confounds and missing-support concerns; they do not make any result equivalent to its input by construction, so the circularity burden remains minimal.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The paper contributes a tool and an HCI evaluation, not a formal derivation. It introduces no numeric free parameters and no speculative physical or formal entities. Its load-bearing assumptions are about design-process theory, the validity of the three-level taxonomy, and the sufficiency of prompt-based generation. The strongest empirical checks are the user study and case studies.

assumptions (4)
  • domain assumption Divergence-before-convergence is a desirable design process and design fixation is a serious problem for multi-agent workflow creation.
    Invoked in Section 1 and Design Goal DG.2; the whole tool is built around encouraging exploration before commitment, but the paper provides only a small formative study as evidence.
  • ad hoc to paper The three-level abstraction recovered by axial coding is a valid organization of the design space.
    Section 3.2.3 and Figure 2; the levels are derived from four interviews and 43 systems, but are not independently validated as the unique or best scaffold. The limitations section notes other scaffolds exist (e.g., Liu et al.'s decision tree).
  • domain assumption Design patterns from prior literature (e.g., reflection, supervision, discussion) transfer to LLM workflows and can be recommended in-situ.
    Section 3.2.2 and Table 1; the paper relies on cited catalogues rather than testing pattern effectiveness itself.
  • domain assumption GPT-4o prompt-based generation is sufficient to create candidate workflows and suggestions for the purpose of evaluating the visualization.
    Section 4.5; the authors acknowledge that better generation methods would improve workflow quality, so the generation step is a simplifying assumption.

how reviews work

0 comments
Cite this review

Pith. "Pith review of FlowForge: Guiding the Creation of Multi-agent Workflows with Design Space Visualization as a Thinking Scaffold." pith.science (2026). https://pith.science/paper/FP4WTP5F

@misc{pith2026250715559,
  author       = {Pith},
  title        = {Pith review of: FlowForge: Guiding the Creation of Multi-agent Workflows with Design Space Visualization as a Thinking Scaffold},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/FP4WTP5F}},
  note         = {Machine review of arXiv:2507.15559}
}
read the original abstract

Multi-agent workflows have become an effective strategy for tackling complicated tasks by decomposing them into multiple sub-tasks and assigning them to specialized agents. However, designing optimal workflows remains challenging due to the vast and intricate design space. Current practices rely heavily on the intuition and expertise of practitioners, often resulting in design fixation or an unstructured, time-consuming exploration of trial-and-error. To address these challenges, this work introduces FLOWFORGE, an interactive visualization tool to facilitate the creation of multi-agent workflow through i) a structured visual exploration of the design space and ii) in-situ guidance informed by established design patterns. Based on formative studies and literature review, FLOWFORGE organizes the workflow design process into three hierarchical levels (i.e., task planning, agent assignment, and agent optimization), ranging from abstract to concrete. This structured visual exploration enables users to seamlessly move from high-level planning to detailed design decisions and implementations, while comparing alternative solutions across multiple performance metrics. Additionally, drawing from established workflow design patterns, FLOWFORGE provides context-aware, in-situ suggestions at each level as users navigate the design space, enhancing the workflow creation process with practical guidance. Use cases and user studies demonstrate the usability and effectiveness of FLOWFORGE, while also yielding valuable insights into how practitioners explore design spaces and leverage guidance during workflow development.

Figures

Figures reproduced from arXiv: 2507.15559 by the authors.

Figure 1
Figure 1. FLOWFORGE facilitates the creation of multi-agent workflows through structured, guided visual exploration of the design space. This is achieved by coordinating a hierarchical tree view (A1) and a scatter plot (A2), both of which use a novel glyph design to represent each workflow’s computational cost and level of abstraction. Users can select a workflow for detailed inspection in the Canvas View (B1-B3), which suppo… view at source ↗
Figure 2
Figure 2. The design space of multi-agent workflows can be conceptualized [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Interface Overview. Users start by entering a task description (A). FLOWFORGE supports workflow creation through two main views: the Design Space View, with a hierarchical tree (B1) and a scatter plot (B2), and the Canvas View, which shows workflow details (C1), in-situ design suggestions (C2), and execution results (C3). 4 SYSTEM FLOWFORGE is an interactive visualization system that supports the design and explorat… view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: Glyphs. (A) At Level 1 (task planning), arcs represent subtasks and node size for sequential count. (B) At Level 2 (agent assignment), bars show agent calls over pseudo-time steps. (C) At Level 3 (agent optimization), workflows are denoted as file icons, as no new stru…
Figure 5
Figure 5. Figure 5: In the Canvas View, the visual representation of each node adapts based on the abstraction level. workflows at the current level or one level above the selected node in the tree. This selective display is informed by the user feedback we received during the iterative d…
Figure 6
Figure 6. Figure 6: Case Study 1: Fast-Forward Video Planning. (A-D) FLOW￾FORGE helps users create multiple creative ways of generating the tran￾script of fast-forward videos for a research paper. (E) In contrast, directly prompting GPT-4o often produces transcripts with conventional stru…
Figure 8
Figure 8. Figure 8: User Study Baseline. We use LangGraph Studio as the baseline system in the user study. Baseline Strongly Disagree (1) Strongly Agree (7) 0% 20% 40% 60% 80% 100% Easy to use Easy to learn Confident in using Useful for design exploration Useful for workflow creation Flow…
Figure 9
Figure 9. Figure 9: User Ratings. Comparison of FLOWFORGE and the baseline system based on post-study questionnaire responses. Participants rated each item on a 7-point Likert scale ranging from strongly disagree (1) to strongly agree (7). Numbers in parentheses indicate the average ratin…
Figure 10
Figure 10. Figure 10: Workflow Creation Time and Quantity. (A) Users spent less time creating their first runnable workflow in FLOWFORGE compared to the baseline. Error bars indicate 95% confidence intervals (CI). (B) Users created more workflows in FLOWFORGE compared with the baseline. su…

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Understanding User Experiences of Computer Use Agents: Design Space and Opportunities for Building Agent UX Prototypes

    cs.HC 2025-10 conditional novelty 6.0 of 10

    Agent prototyping for non-experts requires scaffolds for scoping the agent, designing its chat/UI display, defining user interactions, running it, and debugging its runtime behavior.

Reference graph

Works this paper leans on

63 extracted references · 28 canonical work pages · cited by 1 Pith paper

  1. [57]

    L. Yue, L. Sin Kit, L. Qinghua, Z. Liming, Z. Dehai, X. Xiwei, H. Stefan, and W. Jon. Agent Design Pattern Catalogue: A Collection of Architec- tural Patterns for Foundation Model based Agents.arXiv preprint arXiv: 2405.10467, 2024. doi: 10.48550/arXiv.2405.10467 1, 2, 4, 9

    arXiv doi:10.48550/arxiv.2405.10467 2024

  2. [1]

    Purpose-Built GUI and SDK for AI Development

    Cellum. Purpose-Built GUI and SDK for AI Development. https:// www.vellum.ai/. 2

    work page

  3. [2]

    https://docs.crewai.com/introduction

    CrewAI: Build AI agent teams that work together to tackle complex tasks. https://docs.crewai.com/introduction. 4

    work page

  4. [3]

    The Open-Source Visual AI Programming Environment

    Rivet. The Open-Source Visual AI Programming Environment. https: //rivet.ironcladapp.com/. 2

    work page

  5. [4]

    Multi-Agent Orchestrator framework

    Amazon. Multi-Agent Orchestrator framework. https://awslabs. github.io/multi-agent-orchestrator/. 1, 4

    work page

  6. [5]

    Building Effective Agents

    Anthropic. Building Effective Agents. https://www.anthropic.com/ research/building-effective-agents, 2024. 1, 2, 4

    work page 2024

  7. [6]

    Bostock, V

    M. Bostock, V . Ogievetsky, and J. Heer. D3 Data-Driven Documents.IEEE Transactions on Visualization and Computer Graphics, 17(12):2301–2309,

  8. [8]

    Buxton.Sketching user experiences: getting the design right and the right design

    B. Buxton.Sketching user experiences: getting the design right and the right design. Morgan Kaufmann, San Francisco, 2010. doi: 10.1016/B978 -0-12-374037-3.X5043-3 2

    doi:10.1016/b978 2010

Show all 63 references
  1. [9]

    Cemri, M

    M. Cemri, M. Z. Pan, S. Yang, L. A. Agrawal, B. Chopra, R. Tiwari, K. Keutzer, A. Parameswaran, D. Klein, K. Ramchandran, M. Zaharia, J. E. Gonzalez, and I. Stoica. Why Do Multi-Agent LLM Systems Fail?,

  2. [10]

    C.-M. Chan, W. Chen, Y . Su, J. Yu, W. Xue, S. Zhang, J. Fu, and Z. Liu. ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debate, 2023. doi: 10.48550/arXiv.2308.07201 4

    2023 doi arXiv

  3. [11]

    Dibia, J

    V . Dibia, J. Chen, G. Bansal, S. Syed, A. Fourney, E. Zhu, C. Wang, and S. Amershi. AUTOGEN STUDIO: A No-Code Developer Tool for Building and Debugging Multi-Agent Systems. In D. I. Hernandez Farias, T. Hope, and M. Li, eds.,Proceedings of the 2024 Conference on Empirical Met...

    2024

  4. [12]

    Y . Du, S. Li, A. Torralba, J. B. Tenenbaum, and I. Mordatch. Improving Factuality and Reasoning in Language Models through Multiagent Debate. InProceedings of the 41st International Conference on Machine Learning, ICML’24, article no. 467, 31 pages. JMLR.org, 2024. doi: 10.55...

    2024

  5. [13]

    B. Effect. Manus AI: A General AI Agent. https://manus.im/. Ac- cessed: 2025-03-18. 1

    2025

  6. [14]

    Epperson, G

    W. Epperson, G. Bansal, V . C. Dibia, A. Fourney, J. Gerrits, E. Zhu, and S. Amershi. Interactive debugging and steering of multi-agent ai systems. InProceedings of the 2025 CHI Conference on Human Factors in Computing Systems, pp. 1–15, 2025. 9

    2025

  7. [15]

    Y . Feng, X. Wang, K. K. Wong, S. Wang, Y . Lu, M. Zhu, B. Wang, and W. Chen. PromptMagician: Interactive Prompt Engineering for Text- to-Image Creation .IEEE Transactions on Visualization and Computer Graphics, 30(01):295–305, 2024. doi: 10.1109/TVCG.2023.3327168 2, 3

    2024

  8. [16]

    W. GmbH. React Flow: A library for building node-based UIs. https: //reactflow.dev/, 2024. Accessed: 2025-03-26. 7

    2024

  9. [17]

    Gottweis, W.-H

    J. Gottweis, W.-H. Weng, A. Daryin, T. Tu, A. Palepu, P. Sirkovic, A. Myaskovsky, F. Weissenberger, K. Rong, R. Tanno, et al. Towards an AI co-scientist, 2025. doi: 10.48550/arXiv.2502.18864 1

    2025 doi arXiv

  10. [18]

    Grunde-McLaughlin, M

    M. Grunde-McLaughlin, M. S. Lam, R. Krishna, D. S. Weld, and J. Heer. Designing LLM Chains by Adapting Techniques from Crowdsourcing Workflows.ACM Transactions on Computer-Human Interaction, 32(3), article no. 27, 57 pages, 2025. doi: 10.1145/3716134 1, 2, 4

    2025 doi

  11. [19]

    T. Guo, X. Chen, Y . Wang, R. Chang, S. Pei, N. V . Chawla, O. Wiest, and X. Zhang. Large Language Model based Multi-Agents: A Survey of Progress and Challenges, 2024. doi: 10.48550/arXiv.2402.01680 1, 2, 4, 9

    2024 doi arXiv

  12. [20]

    Heidegger.The Question Concerning Technology, and Other Essays

    M. Heidegger.The Question Concerning Technology, and Other Essays. Harper & Row, New York, 1977. 2

    1977

  13. [21]

    S. Hong, M. Zhuge, J. Chen, X. Zheng, Y . Cheng, J. Wang, C. Zhang, Z. Wang, S. K. S. Yau, Z. Lin, L. Zhou, C. Ran, L. Xiao, C. Wu, and J. Schmidhuber. MetaGPT: Meta Programming for A Multi-Agent Collab- orative Framework. InThe Twelfth International Conference on Learning Rep...

    2024 doi arXiv

  14. [22]

    S. Hu, C. Lu, and J. Clune. Automated Design of Agentic Systems, 2024. doi: 10.48550/arXiv.2408.08435 2, 4, 9

    2024 doi arXiv

  15. [23]

    Z. Jin, Y . Wang, Q. Wang, Y . Ming, T. Ma, and H. Qu. GNNLens: A visual analytics approach for prediction error diagnosis of graph neural networks.IEEE Transactions on Visualization and Computer Graphics, 29(6):3024–3038, 2022. doi: 10.1109/TVCG.2022.3148107 9

    2022

  16. [24]

    LangGraph

    LangChain. LangGraph. Building language agents as graphs. 2, 7

  17. [25]

    LangGraph Studio: The first agent IDE

    LangChain. LangGraph Studio: The first agent IDE. https://blog. langchain.dev/langgraph-studio-the-first-agent-ide/. 8

  18. [26]

    Multi-agent Systems

    LangChain. Multi-agent Systems. https://langchain-ai.github. io/langgraph/concepts/multi_agent/. 4

  19. [27]

    Multi-agent Workflows

    LangChain. Multi-agent Workflows. https://langchain-ai.github. io/langgraph/tutorials/workflows/. 4

  20. [28]

    G. Li, H. A. Al Kader Hammoud, H. Itani, D. Khizbullin, and B. Ghanem. CAMEL: Communicative Agents for "Mind" Exploration of Large Lan- guage Model Society. InProceedings of the 37th International Conference on Neural Information Processing Systems, NIPS ’23, article no. 2264,...

    2023 doi

  21. [29]

    Liang, Z

    T. Liang, Z. He, W. Jiao, X. Wang, Y . Wang, R. Wang, Y . Yang, S. Shi, and Z. Tu. Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debate. In Y . Al-Onaizan, M. Bansal, and Y .-N. Chen, eds.,Proceedings of the 2024 Conference on Empirical Methods in...

    2024 doi

  22. [30]

    M. Liu, J. Shi, Z. Li, C. Li, J. Zhu, and S. Liu. Towards better analysis of deep convolutional neural networks.IEEE Transactions on Visualization and Computer Graphics, 23(1):91–100, 2016. doi: 10.1109/TVCG.2016. 2598831 9

    2016 doi

  23. [31]

    Y . Lu, C. Chen, Y . Chen, K. Huang, M. Zitnik, and Q. Wang. Gnn 101: Visual learning of graph neural networks in your web browser.arXiv preprint arXiv:2411.17849, 2024. doi: 10.48550/arXiv.2411.17849 9

    2024 doi

  24. [32]

    Conversation Patterns

    Microsoft AutoGen. Conversation Patterns. https: //microsoft.github.io/autogen/0.2/docs/tutorial/ conversation-patterns/. 2

  25. [33]

    Multi-Agent Design Patteamrns

    Microsoft AutoGen. Multi-Agent Design Patteamrns. https: //microsoft.github.io/autogen/stable/user-guide/ core-user-guide/design-patterns/intro.html. 2, 4

  26. [34]

    T. Munzner. A nested model for visualization design and validation.IEEE Transactions on Visualization and Computer Graphics, 15(6):921–928,

  27. [35]

    Nakano, J

    R. Nakano, J. Hilton, S. Balaji, J. Wu, L. Ouyang, C. Kim, C. Hesse, S. Jain, V . Kosaraju, W. Saunders, X. Jiang, K. Cobbe, T. Eloundou, G. Krueger, K. Button, M. Knight, B. Chess, and J. Schulman. WebGPT: Browser-assisted question-answering with human feedback, 2022. doi: 10...

    2022 doi arXiv

  28. [36]

    B. Niu, Y . Song, K. Lian, Y . Shen, Y . Yao, K. Zhang, and T. Liu. Flow: Modularized Agentic Workflow Automation, 2025. doi: 10.48550/arXiv. 2501.07834 4

    2025 doi arXiv

  29. [37]

    GPT-4o System Card, 2024

    OpenAI. GPT-4o System Card, 2024. doi: 10.48550/arXiv.2410.21276 6

    2024 doi arXiv

  30. [38]

    J. S. Park, J. O’Brien, C. J. Cai, M. R. Morris, P. Liang, and M. S. Bern- stein. Generative Agents: Interactive Simulacra of Human Behavior. In Proceedings of the 36th Annual ACM Symposium on User Interface Soft- ware and Technology, UIST ’23, article no. 2, 22 pages. ACM, New York,

  31. [39]

    J. R. Penadés, J. Gottweis, L. He, J. B. Patkowski, A. Shurick, W.-H. Weng, T. Tu, A. Palepu, A. Myaskovsky, A. Pawlosky, N. Vivek, K. Alan, and T. R. D. Costa. AI mirrors experimental science to uncover a novel mechanism of gene transfer crucial to bacterial evolution.bioRxiv...

    2025 doi

  32. [40]

    Prasad, A

    A. Prasad, A. Koller, M. Hartmann, P. Clark, A. Sabharwal, M. Bansal, and T. Khot. ADaPT: As-Needed Decomposition and Planning with Language Models. In K. Duh, H. Gomez, and S. Bethard, eds.,Findings of the Association for Computational Linguistics: NAACL 2024, pp. 4226–4252. ...

    2024

  33. [41]

    C. Qian, W. Liu, H. Liu, N. Chen, Y . Dang, J. Li, C. Yang, W. Chen, Y . Su, X. Cong, J. Xu, D. Li, Z. Liu, and M. Sun. ChatDev: Communicative Agents for Software Development. In L.-W. Ku, A. Martins, and V . Sriku- mar, eds.,Proceedings of the 62nd Annual Meeting of the Assoc...

    2024 doi

  34. [42]

    Shinn, F

    N. Shinn, F. Cassano, E. Berman, A. Gopinath, K. Narasimhan, and S. Yao. Reflexion: Language Agents with Verbal Reinforcement Learning, 2023. doi: 10.48550/arXiv.2303.11366 4

    2023 doi arXiv

  35. [43]

    S. Suh, M. Chen, B. Min, T. J.-J. Li, and H. Xia. Luminate: Structured Generation and Exploration of Design Space with Large Language Models for Human-AI Co-Creation. InProceedings of the 2024 CHI Conference on Human Factors in Computing Systems, CHI ’24, article no. 644, 26 p...

    2024

  36. [44]

    S. Suh, B. Min, S. Palani, and H. Xia. Sensecape: Enabling Multilevel Exploration and Sensemaking with Large Language Models. InProceed- ings of the 36th Annual ACM Symposium on User Interface Software and Technology, UIST ’23, article no. 1, 18 pages. ACM, New York, 2023. doi...

    2023

  37. [45]

    Movies dataset

    Vega-Lite. Movies dataset. https://vega.github.io/vega-lite/ data/movies.json, 2017. 7

    2017

  38. [46]

    C. Wang, Q. Wu, and the AG2 Community. AG2: Open-Source Agen- tOS for AI Agents. https://docs.ag2.ai/, 2024. Available at https://docs.ag2.ai/. 4

    2024

  39. [47]

    L. Wang, C. Ma, X. Feng, Z. Zhang, H. Yang, J. Zhang, Z. Chen, J. Tang, X. Chen, Y . Lin, W. X. Zhao, Z. Wei, and J. Wen. A survey on large language model based autonomous agents.Frontiers of Computer Science, 18(6), 2024. doi: 10.1007/s11704-024-40231-1 9

    2024 doi

  40. [49]

    Q. Wang, Y . Ming, Z. Jin, Q. Shen, D. Liu, M. J. Smith, K. Veeramacha- neni, and H. Qu. ATMSeer: Increasing Transparency and Controllability in Automated Machine Learning. InProceedings of the 2019 CHI Con- ference on Human Factors in Computing Systems, CHI ’19, 12 pages, p. ...

    2019

  41. [50]

    Q. Wang, J. Yuan, S. Chen, H. Su, H. Qu, and S. Liu. Visual Genealogy of Deep Neural Networks.IEEE Transactions on Visualization and Computer Graphics, 26(11):3340–3352, 2020. doi: 10.1109/TVCG.2019.2921323 2, 3, 9

    2020

  42. [51]

    Z. Wang, S. Cai, G. Chen, A. Liu, X. Ma, Y . Liang, and T. CraftJarvis. Describe, Explain, Plan and Select: Interactive Planning with Large Lan- guage Models Enables Open-World Multi-Task Agents. InProceedings of the 37th International Conference on Neural Information Processi...

    2023

  43. [52]

    Q. Wu, G. Bansal, J. Zhang, Y . Wu, B. Li, E. Zhu, L. Jiang, X. Zhang, S. Zhang, J. Liu, et al. AutoGen: Enabling next-gen llm applications via multi-agent conversation.arXiv preprint arXiv:2308.08155, 2023. doi: 10. 48550/arXiv.2308.08155 2, 4

    2023 doi arXiv

  44. [53]

    T. Wu, E. Jiang, A. Donsbach, J. Gray, A. Molina, M. Terry, and C. J. Cai. PromptChainer: Chaining Large Language Model Prompts through Visual Programming. InExtended Abstracts of the 2022 CHI Conference on Human Factors in Computing Systems, CHI EA ’22, article no. 359, 10 pa...

    2022

  45. [54]

    T. Wu, M. Terry, and C. J. Cai. AI Chains: Transparent and Controllable Human-AI Interaction by Chaining Large Language Model Prompts. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems, CHI ’22, article no. 385, 22 pages. ACM, New York, 2022. doi:...

    2022

  46. [55]

    T. Wu, H. Zhu, M. Albayrak, A. Axon, A. Bertsch, W. Deng, Z. Ding, B. Guo, S. Gururaja, T.-S. Kuo, J. T. Liang, R. Liu, I. Mandal, J. Milbauer, X. Ni, N. Padmanabhan, S. Ramkumar, A. Sudjianto, J. Taylor, Y .-J. Tseng, P. Vaidos, Z. Wu, W. Wu, and C. Yang. LLMs as Workers in H...

    2025

  47. [56]

    J. Yuan, M. Liu, F. Tian, and S. Liu. Visual Analysis of Neural Archi- tecture Spaces for Summarizing Design Principles.IEEE Transactions on Visualization and Computer Graphics, 29(1):288–298, 2023. doi: 10. 1109/TVCG.2022.3209404 2, 3, 9

    2023

  48. [58]

    Zhang and I

    J. Zhang and I. Arawjo. ChainBuddy: An AI-assisted Agent System for Helping Users Set up LLM Pipelines. InAdjunct Proceedings of the 37th Annual ACM Symposium on User Interface Software and Technology, UIST Adjunct ’24, article no. 48, 3 pages. ACM, New York, 2024. doi: 10 .11...

    2024

  49. [59]

    Zhang, J

    J. Zhang, J. Xiang, Z. Yu, F. Teng, X.-H. Chen, J. Chen, M. Zhuge, X. Cheng, S. Hong, J. Wang, B. Zheng, B. Liu, Y . Luo, and C. Wu. AFlow: Automating Agentic Workflow Generation. InThe Thirteenth International Conference on Learning Representations, 2025. doi: 10. 48550/arXiv...

    2025 doi arXiv

  50. [60]

    Zhang, J

    S. Zhang, J. Zhang, J. Liu, L. Song, C. Wang, R. Krishna, and Q. Wu. Offline Training of Language Model Agents with Functions as Learnable Weights. InProceedings of the 41st International Conference on Machine Learning, ICML’24, article no. 2496, 21 pages. JMLR.org, 2024. 4

    2024

  51. [61]

    Zheng, W.-L

    L. Zheng, W.-L. Chiang, Y . Sheng, S. Zhuang, Z. Wu, Y . Zhuang, Z. Lin, Z. Li, D. Li, E. Xing, et al. Judging LLM-as-a-Judge with MT-Bench and chatbot arena.Advances in Neural Information Processing Systems, 36:46595–46623, 2023. 6

    2023

  52. [62]

    H. Zhou, X. Wan, R. Sun, H. Palangi, S. Iqbal, I. Vuli ´c, A. Korhonen, and S. Ö. Arık. Multi-Agent Design: Optimizing Agents with Better Prompts and Topologies.arXiv preprint arXiv:2502.02533, 2025. doi: 10. 48550/arXiv.2502.02533 2

    2025 doi

  53. [63]

    Zhuge, W

    M. Zhuge, W. Wang, L. Kirsch, F. Faccio, D. Khizbullin, and J. Schmid- huber. GPTSwarm: Language Agents as Optimizable Graphs. In R. Salakhutdinov, Z. Kolter, K. Heller, A. Weller, N. Oliver, J. Scarlett, and F. Berkenkamp, eds.,Proceedings of the 41st International Confer- en...

    2024 doi

  54. [2011]

    doi: 10.1109/TVCG.2011.185 7

    2011 doi

  55. [2025]

    doi: 10.48550/arXiv.2503.13657 1

    doi arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.