Pith. sign in

REVIEW 4 major objections 4 minor 64 references

SHERPA: A Model-Driven Framework for Large Language Model Execution

T0 review · 4 major / 4 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Wrapping LLM execution in hierarchical state machines improves output quality, especially where human best practices exist but training data is scarce.

desk verdict SHERPA is a useful, honest framework paper, but the current evaluation conflates state-machine structure with extra LLM calls, so the headline claim needs stronger baselines. read the letter →

arxiv 2509.00272 v1 pith:FVSCDUGL submitted 2025-08-29 cs.AI cs.SE

Boqi Chen , Kua Chen , José Antonio Hernández López , Gunter Mussbacher , Dániel Varró , Amir Feizpour This is my paper
classification cs.AIcs.SE
keywords statemachineslargelanguagemodelsmodel-drivenengineeringhierarchicalstructuredreasoningbestpracticesLLMagentsworkflowcontrol
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that wrapping large language model calls inside hierarchical state machines—executable diagrams of a task's steps—makes the models' outputs better than direct prompting. The framework, SHERPA, encodes human best practices as states and transitions, with actions on transitions that call LLMs or tools, and policies that choose the next step by rules or by an LLM. Across code generation, class name generation, and scene-graph question answering, SHERPA outperformed a direct baseline in 12 of 15 model-task pairs, with the largest average gain, +6.58 F1 points, in class name generation—a task with well-established modeling practices but little training data. The authors' central claim is that this is exactly the regime where explicit state-machine guidance helps most.

What carries the argument

The central object is the hierarchical state machine, treated as data: states represent sub-tasks, transitions encode the flow prescribed by human best practices, and actions attached to transitions or states call LLMs, tools, or deterministic computations. A policy—rule-based or LLM-based—chooses the next event, while a belief structure stores the trajectory, execution log, and key-value context. The hierarchy decomposes complex tasks into modular sub-tasks, and the decoupling of state-machine definition from action implementation lets engineers change workflow structure without rewriting code.

What would settle it

Repeat the three evaluations matching SHERPA and Direct for number of LLM calls or tokens, and design the state machine using only a training split untouched by the test questions; if Direct matches SHERPA at equal budget or the gains shrink on held-out design, the structure's independent contribution is not established.

Watch

Extended reading notes

Core claim

SHERPA's central claim is that integrating well-designed hierarchical state machines into LLM execution improves output quality over direct prompting, and that the benefit is strongest for tasks with established human best practices but limited training data. The paper demonstrates this on three tasks—code generation, class name generation, and scene-graph question answering—using several LLMs. SHERPA outperformed direct prompting in 12 of 15 model-task pairs, with an average gain of 3.25 points on HumanEval pass@1, 6.58 points on class-name F1, and 2.56 points on CLEVR accuracy. The authors also show that state-machine design is an active variable: different designs trade off performance an

Load-bearing premise

The measured gains are due to the state-machine structure itself, not to the extra LLM calls it makes or to state-machine designs tuned on the evaluation questions.

Editorial extensions

If this is right

  • Well-designed state machines can be dropped into existing LLM workflows; approaches that were previously single-prompt or linear can be re-expressed as state machines and usually improved.
  • State-machine design becomes a real engineering variable: different designs trade performance and cost, and changing designs does not require rewriting action code.
  • For tasks with documented human procedures, smaller LLMs gain the most, suggesting structured execution can narrow the gap between small and large models.
  • Cost reductions are attainable without losing quality: explicitly call-reducing designs such as test-driven generation and preplanned operation sequences maintained performance while lowering LLM invocation counts.
  • The approach gives engineers a handle for predictably budgeting LLM use, since the number of calls is determined by transition structure.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A fair comparison would control inference budget: Direct makes one LLM call per task while SHERPA makes several, so part of the reported gain may be extra compute rather than state-machine structure.
  • Because some state-machine designs were refined based on sampled evaluation questions and iteratively developed with smaller LLMs, the gains could be optimistic for unseen questions; a held-out design split would test this.
  • The separation of state-machine definition from action code suggests a general recipe: encode any domain procedure as a control structure and let an LLM fill in the actions, which could transfer to other under-documented expert tasks.
  • If larger models benefit less from structure on easy tasks, an adaptive design that selects state-machine complexity by model size and task difficulty might restore gains.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. SHERPA is a model-driven framework that structures LLM execution as hierarchical state machines. The SM is defined as data, decoupled from action implementations; a policy (rule-based or LLM-based) selects transitions using the current state and a belief store containing trajectory, action history, and key-value context. LLMs are invoked inside transition/state actions. The paper instantiates SHERPA for three tasks — code generation (test-driven and AgentCoder-style SMs), class-name generation (an inspection-augmented MIG SM), and scene-graph question answering (routing, ReAct-style, and planning SMs) — and compares against Direct prompting on HumanEval, the Modeling dataset, and CLEVR across GPT-4o/Mini, Qwen2.5 7B/72B, and task-specific LLMs. The authors report that SHERPA outperforms Direct in 12/15 cases, with the largest gains in class-name generation, and that SM design choices materially affect both accuracy and the number of LLM calls.

Significance. If the headline claim is supported, SHERPA would be a useful general layer for injecting human best practices into LLM agents, particularly for tasks with established workflows but limited training data. The decoupling of SM design from action implementation is a practical contribution, and the paper usefully shows that existing approaches such as AgentCoder and ReAct can be expressed in the framework. The artifact link and cost measurements are also strengths. However, as presented, the empirical evidence is not yet sufficient to support the central 'significantly improves' claim: the RQ1 comparison conflates SM structure with additional LLM compute, the code-generation Pass@1 metric is applied asymmetrically, and some SM designs appear to be tuned on the evaluation sample. These are load-bearing issues for the paper's headline result, but they are addressable with additional controlled experiments.

major comments (4)
  1. [§V.A.c and §IV.A] The Pass@1 metric is not applied symmetrically. Direct is a single generation ('within 1 attempt'), whereas the test-driven SM retries after test failures and returns the function that passes the most tests after a budget is reached. Thus SHERPA's code-generation result is effectively best-of-N, not Pass@1. The reported +3.25 average improvement in Table I may therefore reflect additional generation attempts rather than SM structure. Please report the distribution of attempts and compare against a best-of-k Direct baseline with a matched number of generated functions, or use Pass@k.
  2. [§V.B.a and Table I] RQ1 compares Direct, a single LLM call, with SHERPA, which is intrinsically multi-step: test generation plus function generation, noun extraction plus classification plus pattern/feedback/inspection, and question classification plus extraction plus counting. Figure 6 shows SHERPA typically uses 3–10 LLM calls. The observed gains (+3.25 Pass@1, +6.58 F1, +2.56 ACC) therefore conflate SM structure with additional inference compute. A non-SM multi-call baseline with the same number of calls (e.g., self-consistency at k, or a linear chain of sub-prompts with identical decomposition) is required to attribute the improvements to the state-machine framework.
  3. [§IV.C and §V.B.c] Parts of the SM design are derived from the evaluation sample, making the comparison partially self-referential. The planning SM uses 'a predefined action sequence as heuristic based on observations from sampled questions', and the SM designs were 'iteratively developed ... with smaller LLMs' during early experiments; the authors acknowledge that 'the final SM design may favor smaller LLMs.' Because the same benchmark questions were used during design and evaluation, the RQ1/RQ2 gains may be inflated by test-set tuning. Please fix SM designs before seeing the test split, or use a separate development/evaluation split for SM design.
  4. [§V.A.b and §V.E] Statistical support is weak. Results are averages over 3 runs at temperature 0.01 with no error bars, confidence intervals, or hypothesis tests. Several Table I differences are small or negative (e.g., GPT-4o on HumanEval: 91.26 vs 90.24; on CLEVR: 90.67 vs 88.33), so the '12/15' and 'significantly improves' statements are not statistically supported. In addition, the 100 CLEVR questions are manually curated and exclusions are described only qualitatively; the final question set is not released, so selection bias cannot be assessed. Please add variance estimates/significance tests and release the exact question set and exclusion criteria.
minor comments (4)
  1. [§V.A.b] The benchmark name is spelled inconsistently as 'Clevr' and 'CLEVR'. Also, 'manually verify each question and exclude any questions...' should be followed by a precise list of exclusion criteria and the final 100-question list.
  2. [Figure 6] The line plots use color as the main differentiator between Direct and SM variants; adding distinct markers or line styles would improve accessibility and readability in grayscale printing.
  3. [§III.B] The policy definition π(s,Ts,B) → e omits the data parameters that actions require. The text mentions 'the selected event along with any data required by the transitions,' but the formal notation and the prompt template would be clearer if the data/parameter passing were made explicit.
  4. [References] Reference [48] is a GitHub page accessed in 2025. Please cite a stable archival source or include the exact prompt template in the appendix/artifact, since the Direct code-generation baseline depends on it.

Circularity Check

0 steps flagged · score 2.0 of 10

No equation-level circularity; central empirical claim is supported by external benchmarks, with mild self-citation and self-reported test-set influence in secondary comparisons.

full rationale

SHERPA is an empirical framework paper, not a formal derivation. The central claim — that integrating state machines improves LLM outputs — is tested by comparing SM-based agents to Direct prompting on three external benchmarks (HumanEval, Modeling, CLEVR). The RQ1 SMs (test-driven, inspection, routing) are grounded in established practices (TDD), the published AgentCoder/ReAct approaches, and a standard routing strategy; they are not constructed from the evaluation data. The class-name inspection SM is motivated by the authors' own MIG approach [40], a self-citation, but MIG is a published prior method and is not invoked as a uniqueness theorem or external proof; the self-citation is not load-bearing for the main result. The paper does disclose two design-influence limitations: the planning SM is 'based on observations from sampled questions' (Section IV.C), and 'we iteratively developed the SM with smaller LLMs' (Section V.B.c). These are genuine validity threats for the secondary RQ2 planning-SM comparison and the RQ1.2 size-effect conclusion, respectively. However, they are not fitted parameters renamed as predictions: the planning SM is one of several configurations, and the main RQ1 claim does not reduce to it. The inference-budget confound (Direct uses one LLM call while SHERPA uses several) is a correctness/validity concern, not a circularity of the derivation-chain kind defined here. No equation or fitted parameter is defined in terms of the claimed result; therefore the paper has no significant circularity, only mild self-reference and evaluation-design caveats that a score of 2 reflects.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

No numeric parameters are fitted to data, but two design-level choices function as free parameters: the planning SM heuristic (explicitly tuned on the sampled CLEVR questions) and the overall SM designs (iteratively developed with smaller LLMs). The framework relies on the domain assumptions listed above.

free parameters (3)
  • Planning SM action sequence heuristic = derived from sampled CLEVR questions
    Section IV.C states the predefined action sequence is 'based on observations from sampled questions'; this is manual tuning on the evaluation set.
  • State machine designs per task = test-driven, agent coder, inspection, MIG, routing, ReAct, planning SMs
    Designs chosen by authors, iteratively developed with smaller LLMs (Section V.B.c); the choice influences RQ1 and RQ2 outcomes.
  • Maximum number of state transitions = 10
    Set in Section V.A.d; acts as an early-stop budget that affects success when retries are needed.
assumptions (3)
  • domain assumption State machines can faithfully encode human best practices for the target tasks.
    The whole method assumes that expert workflows can be represented as hierarchical state machines (Section III.A).
  • domain assumption LLM-based policies can reliably choose the next transition given current state and belief.
    The policy prompt (Figure 3) asks the LLM to select transitions; if routing errors are frequent, the SM can misdirect execution.
  • domain assumption Benchmark datasets (HumanEval, Modeling, CLEVR) are representative proxies for the target tasks.
    Used as evaluation in Section V.A.b; HumanEval pass@1 may not guarantee correctness, as the paper itself notes in Section V.E.

how reviews work

0 comments
Cite this review

Pith. "Pith review of SHERPA: A Model-Driven Framework for Large Language Model Execution." pith.science (2026). https://pith.science/paper/FVSCDUGL

@misc{pith2026250900272,
  author       = {Pith},
  title        = {Pith review of: SHERPA: A Model-Driven Framework for Large Language Model Execution},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/FVSCDUGL}},
  note         = {Machine review of arXiv:2509.00272}
}
read the original abstract

Recently, large language models (LLMs) have achieved widespread application across various fields. Despite their impressive capabilities, LLMs suffer from a lack of structured reasoning ability, particularly for complex tasks requiring domain-specific best practices, which are often unavailable in the training data. Although multi-step prompting methods incorporating human best practices, such as chain-of-thought and tree-of-thought, have gained popularity, they lack a general mechanism to control LLM behavior. In this paper, we propose SHERPA, a model-driven framework to improve the LLM performance on complex tasks by explicitly incorporating domain-specific best practices into hierarchical state machines. By structuring the LLM execution processes using state machines, SHERPA enables more fine-grained control over their behavior via rules or decisions driven by machine learning-based approaches, including LLMs. We show that SHERPA is applicable to a wide variety of tasks-specifically, code generation, class name generation, and question answering-replicating previously proposed approaches while further improving the performance. We demonstrate the effectiveness of SHERPA for the aforementioned tasks using various LLMs. Our systematic evaluation compares different state machine configurations against baseline approaches without state machines. Results show that integrating well-designed state machines significantly improves the quality of LLM outputs, and is particularly beneficial for complex tasks with well-established human best practices but lacking data used for training LLMs.

Figures

Figures reproduced from arXiv: 2509.00272 by the authors.

Figure 2
Figure 2. shows the metamodel of the SM used in SHERPA. <<abstract>> State * name description Transition * Action * name SherpaStateMachine Tag * name CompositeState SimpleState Parameter name description type 2..* 1 source 1 target * Condition * expression * * on_exit * actions * on_enter * Event * name 1 [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 4
Figure 4. Example of SHERPA for the scene graph-based question answering selecting the next transition within the context of question an￾swering. Upon policy initialization, a task description defining the overall task, along with an output instruction specifying the expected output, is provided. During execution, when the LLM-based policy is invoked, the relevant execution history, including the trajectory and previous actio… view at source ↗
Figure 5
Figure 5. SM design for the use cases; states for outputs and actions on the transitions are omitted for conciseness [PITH_FULL_IMAGE:figures/full_fig_p006_5.png] view at source ↗
Figures from the paper (1 more)
Figure 6
Figure 6. Figure 6: Comparison of different SM designs and the [PITH_FULL_IMAGE:figures/full_fig_p009_6.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

64 extracted references · 48 canonical work pages

  1. [1]

    Statecharts: a visual formalism for complex systems,

    D. Harel, “Statecharts: a visual formalism for complex systems,” Science of Computer Programming , vol. 8, no. 3, pp. 231–274, 1987

    work page 1987

  2. [2]

    The Unified Modeling Language reference manual,

    I. Jacobson and J. R. G. Booch, “The Unified Modeling Language reference manual,” Pearson, 2021

    work page 2021

  3. [3]

    Language models are few-shot learners,

    T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell et al., “Language models are few-shot learners,” in Advances in Neural Information Processing Systems, vol. 33, 2020, pp. 1877–1901

    work page 2020

  4. [4]

    GPT-4o system card,

    A. Hurst, A. Lerer, A. P. Goucher, A. Perelman, A. Ramesh, A. Clark, A. Ostrow, A. Welihinda, A. Hayes, A. Radford et al., “GPT-4o system card,” arXiv preprint arXiv:2410.21276 , 2024

    arXiv 2024

  5. [5]

    Qwen2.5 technical report,

    A. Yang, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Li, D. Liu, F. Huang, H. Wei et al. , “Qwen2.5 technical report,” arXiv preprint arXiv:2412.15115, 2024

    arXiv 2024

  6. [6]

    A survey on large language models for code generation,

    J. Jiang, F. Wang, J. Shen, S. Kim, and S. Kim, “A survey on large language models for code generation,” arXiv preprint arXiv:2406.00515 , 2024

    arXiv 2024

  7. [7]

    Survey of large language models for answering questions across various fields,

    P. Shailendra, R. C. Ghosh, R. Kumar, and N. Sharma, “Survey of large language models for answering questions across various fields,” in 2024 10th International Conference on Advanced Computing and Communication Systems (ICACCS) , vol. 1. IEEE, 2024, pp. 520–527

    work page 2024

  8. [8]

    Understanding the planning of LLM agents: A survey,

    X. Huang, W. Liu, X. Chen, X. Wang, H. Wang, D. Lian, Y . Wang, R. Tang, and E. Chen, “Understanding the planning of LLM agents: A survey,” arXiv preprint arXiv:2402.02716 , 2024

    arXiv 2024

Show all 64 references
  1. [9]

    A survey on large language models for automated planning,

    M. Aghzal, E. Plaku, G. J. Stein, and Z. Yao, “A survey on large language models for automated planning,” arXiv preprint arXiv:2502.12435 , 2025

    2025 arXiv

  2. [10]

    On the use of large language models in model-driven engineering,

    J. Di Rocco, D. Di Ruscio, C. Di Sipio, P. T. Nguyen, and R. Rubei, “On the use of large language models in model-driven engineering,” Software and Systems Modeling , pp. 1–26, 2025

    2025

  3. [11]

    A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions,

    L. Huang, W. Yu, W. Ma, W. Zhong, Z. Feng, H. Wang, Q. Chen, W. Peng, X. Feng, B. Qin et al. , “A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions,” ACM Transactions on Information Systems , vol. 43, no. 2, pp. 1–55, 2025

    2025

  4. [12]

    Retrieval- augmented generation for knowledge-intensive NLP tasks,

    P. Lewis, E. Perez, A. Piktus, F. Petroni, V . Karpukhin, N. Goyal, H. K ¨uttler, M. Lewis, W.-t. Yih, T. Rockt ¨aschel et al. , “Retrieval- augmented generation for knowledge-intensive NLP tasks,” in Advances in Neural Information Processing Systems , vol. 33, 2020, pp. 9459–9474

    2020

  5. [13]

    Self-RAG: Learning to retrieve, generate, and critique through self-reflection,

    A. Asai, Z. Wu, Y . Wang, A. Sil, and H. Hajishirzi, “Self-RAG: Learning to retrieve, generate, and critique through self-reflection,” in NeurIPS 2023 Workshop on Instruction Tuning and Instruction Following , 2023

    2023

  6. [14]

    Active retrieval augmented generation,

    Z. Jiang, F. F. Xu, L. Gao, Z. Sun, Q. Liu, J. Dwivedi-Yu, Y . Yang, J. Callan, and G. Neubig, “Active retrieval augmented generation,” in Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, 2023, pp. 7969–7992

    2023

  7. [15]

    Demonstrate-Search-Predict: Composing retrieval and language models for knowledge-intensive nlp,

    O. Khattab, K. Santhanam, X. L. Li, D. Hall, P. Liang, C. Potts, and M. Zaharia, “Demonstrate-Search-Predict: Composing retrieval and language models for knowledge-intensive nlp,” arXiv preprint arXiv:2212.14024, 2022

    2022 arXiv

  8. [16]

    Chain of thought prompting elicits reasoning in large language models,

    J. Wei, X. Wang, D. Schuurmans, M. Bosma, brian ichter, F. Xia, E. H. Chi, Q. V . Le, and D. Zhou, “Chain of thought prompting elicits reasoning in large language models,” in Advances in Neural Information Processing Systems, 2022

    2022

  9. [17]

    Tree of thoughts: Deliberate problem solving with large language models,

    S. Yao, D. Yu, J. Zhao, I. Shafran, T. Griffiths, Y . Cao, and K. Narasimhan, “Tree of thoughts: Deliberate problem solving with large language models,” in Advances in Neural Information Processing Systems , 2023

    2023

  10. [18]

    ReAct: Synergizing reasoning and acting in language models,

    S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. Narasimhan, and Y . Cao, “ReAct: Synergizing reasoning and acting in language models,” in International Conference on Learning Representations , 2023

    2023

  11. [19]

    StateFlow: Enhancing LLM task-solving through state-driven workflows,

    Y . Wu, T. Yue, S. Zhang, C. Wang, and Q. Wu, “StateFlow: Enhancing LLM task-solving through state-driven workflows,” in First Conference on Language Modeling , 2024

    2024

  12. [20]

    SMoT: Think in state machine,

    J. Liu and J. Shuai, “SMoT: Think in state machine,” arXiv preprint arXiv:2312.17445, 2023

    2023 arXiv

  13. [21]

    Qwen2. 5-Coder technical report,

    B. Hui, J. Yang, Z. Cui, J. Yang, D. Liu, L. Zhang, T. Liu, J. Zhang, B. Yu, K. Lu et al., “Qwen2. 5-Coder technical report,” arXiv preprint arXiv:2409.12186, 2024

    2024 arXiv

  14. [22]

    The Llama 3 herd of models,

    A. Grattafiori, A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Vaughan et al., “The Llama 3 herd of models,” arXiv preprint arXiv:2407.21783 , 2024

    2024 arXiv

  15. [23]

    Attention is all you need,

    A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” Advances in Neural Information Processing Systems , 2017

    2017

  16. [24]

    Instruction tuning for large language models: A survey,

    S. Zhang, L. Dong, X. Li, S. Zhang, X. Sun, S. Wang, J. Li, R. Hu, T. Zhang, F. Wu et al., “Instruction tuning for large language models: A survey,” arXiv preprint arXiv:2308.10792 , 2023

    2023

  17. [25]

    Training language models to follow instructions with human feedback,

    L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray et al. , “Training language models to follow instructions with human feedback,” Advances in Neural Information Processing Systems , vol. 35, pp. 27 730–27 744, 2022

    2022

  18. [26]

    ToolLLM: Facilitating large language models to master 16000+ real-world APIs,

    Y . Qin, S. Liang, Y . Ye, K. Zhu, L. Yan, Y . Lu, Y . Lin, X. Cong, X. Tang, B. Qian et al., “ToolLLM: Facilitating large language models to master 16000+ real-world APIs,” in International Conference on Learning Representations, 2024

    2024

  19. [27]

    Automated domain modeling with large language models: A comparative study,

    K. Chen, Y . Yang, B. Chen, J. A. H. L´opez, G. Mussbacher, and D. Varr´o, “Automated domain modeling with large language models: A comparative study,” in 2023 ACM/IEEE 26th International Conference on Model Driven Engineering Languages and Systems (MODELS) . IEEE, 2023, pp. 162–172

    2023

  20. [28]

    LangGraph: Building knowledge graphs with language models,

    “LangGraph: Building knowledge graphs with language models,” https: //langchain-ai.github.io/langgraph/, accessed: 2024-09-30

    2024

  21. [29]

    MemGPT: Towards LLMs as operating systems,

    C. Packer, V . Fang, S. G. Patil, K. Lin, S. Wooders, and J. E. Gonzalez, “MemGPT: Towards LLMs as operating systems,” arXiv preprint arXiv:2310.08560, 2023

    2023 arXiv

  22. [30]

    A method for synthesizing sequential circuits,

    G. H. Mealy, “A method for synthesizing sequential circuits,” The Bell System Technical Journal, vol. 34, no. 5, pp. 1045–1079, 1955

    1955

  23. [31]

    A systematic review of code generation proposals from state machine specifications,

    E. Domı, B. P ´erez, ´A. L. Rubio et al. , “A systematic review of code generation proposals from state machine specifications,” Information and Software Technology, vol. 54, no. 10, pp. 1045–1066, 2012

    2012

  24. [32]

    Complete code generation from UML state machine,

    A. R. Van Cam Pham, S. G ´erard, and S. Li, “Complete code generation from UML state machine,” in Proceedings of the 5th International Conference on Model-Driven Engineering and Software Development , vol. 1, 2017, pp. 208–219

    2017

  25. [33]

    Model checking of hierarchical state machines,

    R. Alur and M. Yannakakis, “Model checking of hierarchical state machines,” ACM SIGSOFT Software Engineering Notes , vol. 23, no. 6, pp. 175–188, 1998

    1998

  26. [34]

    PlantUML at a glance,

    “PlantUML at a glance,” https://plantuml.com/

  27. [35]

    Umple: Model-driven development for open source and education,

    T. C. Lethbridge, A. Forward, O. Badreddin, D. Brestovansky, M. Garzon, H. Aljamaan, S. Eid, A. H. Orabi, M. H. Orabi, V . Abdelzad et al. , “Umple: Model-driven development for open source and education,” Science of Computer Programming , vol. 208, p. 102665, 2021

    2021

  28. [36]

    R. S. Sutton, A. G. Barto et al., Reinforcement learning: An Introduction. MIT press Cambridge, 1998

    1998

  29. [37]

    Large language models for software engi- neering: A systematic literature review,

    X. Hou, Y . Zhao, Y . Liu, Z. Yang, K. Wang, L. Li, X. Luo, D. Lo, J. Grundy, and H. Wang, “Large language models for software engi- neering: A systematic literature review,” ACM Transactions on Software Engineering and Methodology , vol. 33, no. 8, pp. 1–79, 2024

    2024

  30. [38]

    A survey of large language model agents for question answering,

    M. Yue, “A survey of large language model agents for question answering,” arXiv preprint arXiv:2503.19213 , 2025

    2025 arXiv

  31. [39]

    AgentCoder: Multi-agent-based code generation with iterative testing and optimisation,

    D. Huang, J. M. Zhang, M. Luck, Q. Bu, Y . Qing, and H. Cui, “AgentCoder: Multi-agent-based code generation with iterative testing and optimisation,” arXiv preprint arXiv:2312.13010 , 2023

    2023 arXiv

  32. [40]

    Multi-step iterative automated domain modeling with large language models,

    Y . Yang, B. Chen, K. Chen, G. Mussbacher, and D. Varr ´o, “Multi-step iterative automated domain modeling with large language models,” in Proceedings of the ACM/IEEE 27th International Conference on Model Driven Engineering Languages and Systems , ser. MODELS Companion ’24. N...

    2024

  33. [41]

    Doing more with less–implementing routing strategies in large language model-based systems: An extended survey,

    C. Varangot-Reille, C. Bouvard, A. Gourru, M. Ciancone, M. Schaeffer, and F. Jacquenet, “Doing more with less–implementing routing strategies in large language model-based systems: An extended survey,” arXiv preprint arXiv:2502.00409, 2025

    2025 arXiv

  34. [42]

    GPT-4o mini: advancing cost-efficient intelligence,

    OpenAI, “GPT-4o mini: advancing cost-efficient intelligence,” https: //openai.com/index/gpt-4o-mini-advancing-cost-efficient-intelligence/

  35. [43]

    Evaluating large language models trained on code,

    M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. D. O. Pinto, J. Kaplan, H. Edwards, Y . Burda, N. Joseph, G. Brockman et al., “Evaluating large language models trained on code,” arXiv preprint arXiv:2107.03374 , 2021

    2021 arXiv

  36. [44]

    CLEVR: A diagnostic dataset for compositional language and elementary visual reasoning,

    J. Johnson, B. Hariharan, L. Van Der Maaten, L. Fei-Fei, C. Lawrence Zit- nick, and R. Girshick, “CLEVR: A diagnostic dataset for compositional language and elementary visual reasoning,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , 2017, p...

    2017

  37. [45]

    Inferring and executing programs for visual reasoning,

    J. Johnson, B. Hariharan, L. Van Der Maaten, J. Hoffman, L. Fei- Fei, C. Lawrence Zitnick, and R. Girshick, “Inferring and executing programs for visual reasoning,” in Proceedings of the IEEE International Conference on Computer Vision , 2017, pp. 2989–2998

    2017

  38. [46]

    Embedding- based automated assessment of domain models,

    K. Chen, B. Chen, Y . Yang, G. Mussbacher, and D. Varr ´o, “Embedding- based automated assessment of domain models,” in Proceedings of the ACM/IEEE 27th International Conference on Model Driven Engineering Languages and Systems , 2024, pp. 87–94

    2024

  39. [47]

    Artifact-sherpa4modeling

    B. Chen, K. Chen, J. A. H. L ´opez, G. Mussbacher, D. Varr ´o, and A. Feizpour, “Artifact-sherpa4modeling.” [Online]. Available: https://zenodo.org/records/16338136

  40. [48]

    Performance of ChatGPT on HumanEval,

    “Performance of ChatGPT on HumanEval,” https://github.com/ saschaschramm/chatgpt?tab=readme-ov-file#performance, accessed: 2025-03-23

    2025

  41. [49]

    On Codex prompt engineering for OCL generation: an empirical study,

    S. Abukhalaf, M. Hamdaqa, and F. Khomh, “On Codex prompt engineering for OCL generation: an empirical study,” in 2023 IEEE/ACM 20th International Conference on Mining Software Repositories (MSR) . IEEE, 2023, pp. 148–157

    2023

  42. [50]

    Model-driven prompt engineering,

    R. Claris ´o and J. Cabot, “Model-driven prompt engineering,” in 2023 ACM/IEEE 26th International Conference on Model Driven Engineering Languages and Systems (MODELS) . IEEE, 2023, pp. 47–54

    2023

  43. [51]

    Automating the development of task-oriented llm-based chatbots,

    J. S. Cuadrado, S. P´erez-Soler, E. Guerra, and J. de Lara, “Automating the development of task-oriented llm-based chatbots,” in ACM Conversational User Interfaces 2024, CUI 2024, Luxembourg, July 8-10, 2024 . ACM, 2024, p. 11

    2024

  44. [52]

    Chainbuddy: An ai-assisted agent system for generating LLM pipelines,

    J. Zhang and I. Arawjo, “Chainbuddy: An ai-assisted agent system for generating LLM pipelines,” in Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems, CHI 2025, YokohamaJapan, 26 April 2025- 1 May 2025 . ACM, 2025, pp. 241:1–241:21

    2025

  45. [53]

    A DSL for testing LLMs for fairness and bias,

    S. Morales, R. Claris ´o, and J. Cabot, “A DSL for testing LLMs for fairness and bias,” in Proceedings of the ACM/IEEE 27th International Conference on Model Driven Engineering Languages and Systems , 2024, pp. 203–213

    2024

  46. [54]

    Towards using few-shot prompt learning for automating model completion,

    M. B. Chaaben, L. Burgue ˜no, and H. Sahraoui, “Towards using few-shot prompt learning for automating model completion,” in 2023 IEEE/ACM 45th international conference on software engineering: New ideas and emerging results (ICSE-NIER) . IEEE, 2023, pp. 7–12

    2023

  47. [55]

    Text2VQL: teaching a model query language to open-source language models with ChatGPT,

    J. A. H. L ´opez, M. F ¨oldi´ak, and D. Varr ´o, “Text2VQL: teaching a model query language to open-source language models with ChatGPT,” in Proceedings of the ACM/IEEE 27th International Conference on Model Driven Engineering Languages and Systems , 2024, pp. 13–24

    2024

  48. [56]

    On the assessment of generative AI in modeling tasks: an experience report with ChatGPT and UML,

    J. C ´amara, J. Troya, L. Burgue ˜no, and A. Vallecillo, “On the assessment of generative AI in modeling tasks: an experience report with ChatGPT and UML,” Software and Systems Modeling , vol. 22, no. 3, pp. 781–793, 2023

    2023

  49. [57]

    Reflexion: Language agents with verbal reinforcement learning,

    N. Shinn, F. Cassano, A. Gopinath, K. Narasimhan, and S. Yao, “Reflexion: Language agents with verbal reinforcement learning,” in Advances in Neural Information Processing Systems , 2024

    2024

  50. [58]

    AutoGen: Enabling Next-Gen LLM applications via multi- agent conversation framework,

    Q. Wu, G. Bansal, J. Zhang, Y . Wu, B. Li, E. Zhu, L. Jiang, X. Zhang, S. Zhang, J. Liu, A. H. Awadallah, R. W. White, D. Burger, and C. Wang, “AutoGen: Enabling Next-Gen LLM applications via multi- agent conversation framework,” in First Conference on Language Modeling, 2024

    2024

  51. [59]

    MetaGPT: Meta programming for a multi-agent collaborative framework,

    S. Hong, M. Zhuge, J. Chen, X. Zheng, Y . Cheng, J. Wang, C. Zhang, Z. Wang, S. K. S. Yau, Z. Lin, L. Zhou, C. Ran, L. Xiao, C. Wu, and J. Schmidhuber, “MetaGPT: Meta programming for a multi-agent collaborative framework,” in International Conference on Learning Representations, 2024

    2024

  52. [60]

    Encouraging divergent thinking in large language models through multi-agent debate,

    T. Liang, Z. He, W. Jiao, X. Wang, Y . Wang, R. Wang, Y . Yang, Z. Tu, and S. Shi, “Encouraging divergent thinking in large language models through multi-agent debate,” in Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing , 2024, pp. 17 889– 17 904

    2024

  53. [61]

    LLM-SAP: Large language models situational awareness-based planning,

    L. Wang and H. Zhong, “LLM-SAP: Large language models situational awareness-based planning,” in 2024 IEEE International Conference on Multimedia and Expo Workshops (ICMEW) , 2024, pp. 1–6

    2024

  54. [62]

    CrewAI: Fast and flexible multi-agent automation framework,

    CrewAI Inc., “CrewAI: Fast and flexible multi-agent automation framework,” https://github.com/crewAIInc/crewAI, 2025, accessed: 2025-07-24

    2025

  55. [63]

    n8n: Secure workflow automation for technical teams,

    n8n Contributors, “n8n: Secure workflow automation for technical teams,” https://github.com/n8n-io/n8n, 2025, accessed: 2025-07-24

    2025

  56. [64]

    Haystack: The production-ready open secure ai framework,

    deepset-ai, “Haystack: The production-ready open secure ai framework,” urlhttps://github.com/deepset-ai/haystack, 2025, accessed: 2025-07-24

    2025

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.