Pith. sign in

REVIEW 5 major objections 7 minor 46 references

Role-aligned LLM agents under human oversight can execute an ERP's operational decisions end-to-end, not merely record them.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 18:17 UTC pith:UWVL23VX

load-bearing objection A coherent reference architecture for LLM agents on an ERP backend with an honest limitations section, but the headline feasibility claim rests on a single simulated seed and the abstract overstates what the experiments support. the 5 major comments →

arxiv 2607.17331 v1 pith:UWVL23VX submitted 2026-07-19 cs.AI cs.MA

Agentic ERP: Multi-Agent Large Language Model Architecture for Autonomous Enterprise Resource Planning

Zhihao Liu , Tianyu Wang , Xi Vincent Wang , Lihui Wang This is my paper
classification cs.AI cs.MA
keywords expert systemslarge language model agentsenterprise resource planningmulti-agent orchestrationhuman-in-the-loop automationdecision supportautonomous ERProle-based decomposition
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper sets out to show that an enterprise resource planning system—the software a company uses to record sales, inventory, purchases, and finances—can move from passively storing transactions to actively making the routine decisions those transactions depend on. It claims that a small population of large-language-model agents, each confined to one business role such as sales, inventory, purchasing, finance, or coordination, can decide when to reorder, which orders to accept, and how to absorb supplier delays, as long as every write is gated by a risk-tiered human approval harness. The evidence is a synthetic year-long simulation of a tea-trading company in which the agentic system serviced all 639 orders with zero stockouts, while a rule-based automation baseline under the same demand stream accumulated hundreds of stockouts. If the claim holds, the bottleneck to more autonomous enterprise software is not raw model ability but verified trust, and the paper offers a reference architecture and an evaluation protocol for testing that; the authors themselves present the result as feasibility on one simulated SME, with quality judged by automated LLM-as-judge grading rather than human raters.

Core claim

The paper's central claim is that autonomous ERP operation can be formulated as a constrained sequential-decision problem and made tractable by a role-decomposed population of LLM agents coordinated by a graph-based Planner–Executor–Reflector–Responder pipeline. It argues a decomposition bound: if a router picks the correct agent with error ε_r and each agent chooses among a small tool set of average size k̄, the total per-step tool-selection error is at most ε_r + (1 − ε_r)ε_sel(k̄), far below a monolithic agent choosing among all 46 tools. The architecture externalizes grading criteria and sprint contracts so generation is separated from evaluation, and every write passes through risk tier

What carries the argument

The load-bearing device is the role-based decomposition bound (equation 4), which expresses multi-agent tool-selection error as routing error plus per-agent selection error at the mean tool-set size. It turns "assign each agent a small, role-coherent tool set" into a measured property: five role-aligned agents choosing among an average of about 9 tools each should mis-select far less often than one agent choosing among all 46 tools. The other two carriers are the graph-based orchestration, whose Reflector returns a specific failure reason rather than a bare boolean so replanning is conditioned on why the previous attempt failed, and the risk-tiered write harness, which blocks every unapprove

Load-bearing premise

The load-bearing premise is that one hand-configured simulation—a single demand-and-supply realization with a fixed 10% supplier-delay probability and a rule-based baseline frozen at reorder point 50, order quantity 100—is representative enough to support the operational gap, and that an LLM's tool-selection error really does grow monotonically with the number of candidate tools as the decomposition bound assumes.

What would settle it

Re-run the 365-day protocol across multiple demand seeds, supplier-delay distributions, and baseline policies, including a rule-based controller whose reorder parameters are tuned to the observed demand. If a decently tuned rule-based policy or any single seed shows the agentic system stocking out in meaningful numbers, or if the 0-versus-302 stockout gap narrows dramatically, the operational claim fails. The decomposition bound can also be tested directly by measuring tool-selection error at different tool-set sizes and checking whether the gap follows the predicted growing trend.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • If the claim holds, an ERP backend can handle routine operational decisions—reordering, credit checks, order acceptance, delivery-date negotiation—through role-aligned LLM agents, with human review reserved for high- and critical-risk actions.
  • The decomposition result predicts that as the global tool space grows, the accuracy gap between role-aligned multi-agent and monolithic single-agent designs widens, so the architecture should stay competitive as ERP modules are added.
  • Externalizing grading criteria and sprint contracts makes automation inspectable: workflows can be audited, acceptance criteria revised per task, and failures traced to a specific step, countering the self-evaluation bias of an agent grading its own output.
  • The benchmark suggests the decisive factor is a reflective multi-agent structure rather than any specific orchestration library; the proposed graph matches the strongest conversational multi-agent methods and beats single-step reasoning.
  • Sustained simulation shows the main value of agentic reasoning is operational quality (zero stockouts vs. hundreds), with a smaller financial advantage (+10.7% ending cash over the rule-based baseline), because financial outcomes are downstream aggregates bounded by shared demand and margins.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the role-decomposition logic is general: in any high-stakes domain where an LLM selects among a large tool or action space, splitting that space by human roles should reduce selection error and context cost—so the mechanism could transfer to clinical ordering, logistics control, or grid operations.
  • Editorial inference: the headline gap is a single-realization result; a skeptical reader should ask for multi-seed runs and for a rule-based baseline whose reorder parameters are tuned to the demand regime, since either could shrink the 0-versus-302 stockout margin.
  • Editorial inference: a concrete testable extension is to run the same 365-day protocol on the second electronics-distribution fixture the paper mentions but does not exercise, and to compare adaptive versus tuned-static inventory policies over full demand-seed sweeps.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

5 major / 7 minor

Summary. The paper proposes Agentic ERP, a multi-agent LLM architecture for autonomous execution of operational ERP decisions. It formalizes the problem as a constrained sequential decision process over an enterprise state, introduces role-aligned agents (Coordinator, Sales, Inventory, Purchasing, Finance) with small per-agent tool sets, and wraps them in a Planner-Executor-Reflector-Responder orchestration with a risk-tiered human-in-the-loop harness. The evaluation has five components: E1 (36 scenario tasks), E2 (six orchestration paradigms on 15 crisis scenarios), E3 (365-day agent-in-the-loop simulation vs. RPA and no-intervention baselines), E4 (ablation), and E5 (harness enhancement). The authors claim that the architecture sustains a simulated year with zero stockouts while RPA incurs 302, that the multi-agent design beats single-agent on tool-selection accuracy and efficiency, and that the proposed orchestration is competitive with the best baselines while providing inspectability and auditability.

Significance. If the empirical claims hold, the paper makes a useful contribution: it is one of the first to demonstrate a concrete reference architecture that combines LLM reasoning with a real ERP backend (Odoo), role-based decomposition, externalized grading contracts, and a human-in-the-loop safety gate. The decomposition argument in Eq. (4), the reflection-loop reasoning around Eq. (5), and the three-layer evaluation protocol are valuable intellectual scaffolding. The paper is also unusually candid about its limitations, explicitly noting single-seed main-table runs and the lack of human calibration of the LLM-as-judge. However, the central feasibility claim rests on a single stochastic simulation trajectory with hand-set parameters and no sensitivity analysis, and the orchestration comparison shows only one statistically significant pairwise win after multiple-comparison correction. These issues are load-bearing because the abstract and conclusions use 'significantly better' and 'zero stockouts' as headline evidence. With additional seeds, sensitivity analysis, and more careful claim calibration, the framework would be a credible and reproducible template for autonomous ERP research.

major comments (5)
  1. [§5.2, Tables 11–13] The central E3 result — 0 stockouts vs. 302 for RPA and +10.7% cash — is reported for a single stochastic realization with hand-set parameters: 10% supplier-delay probability, 95% on-time payment, and an RPA baseline fixed at (s=50, Q=100). No seed variation, no demand-distribution perturbation, and no RPA parameter sweep are reported, and no code or data are released. Because 'zero stockouts' is a binary per-trajectory outcome, one favorable draw can produce it; under a demand stream with different variance or a baseline with a tuned reorder policy, the gap could narrow or disappear. I ask the authors to report at least 10–20 seeds for the full E3 comparison, with distributions of stockouts and ending cash, and to perform a sensitivity analysis over demand coefficient of variation, supplier-delay probability, and RPA (s,Q) parameters. In addition, the comparison is not fully apples-to-a
  2. [§5.4, Table 18] The E2 evidence for orchestration superiority is much weaker than the abstract suggests. After Holm–Bonferroni correction, the only significant pairwise advantage of LangGraph over a baseline is versus ReAct (pHolm = 0.040). LangGraph is statistically indistinguishable from CrewAI (p=0.554) and Plan-and-Solve (p=0.376), and only borderline against Function Calling (p=0.115). AutoGen is numerically higher (S=0.717 vs. 0.644) with a bootstrap CI excluding zero, although not significant after correction (p=0.127). The abstract's phrase 'significantly better than the baseline' is therefore not supported as a general claim across orchestration baselines; it is supported only against single-step reasoning and, in E3, against a specific fixed-rule RPA policy. I recommend rewording the abstract and Section 6 to state precisely which baseline each significant claim refers to, and to frame the E2
  3. [§3.2, Eq. (4)] The decomposition bound is essentially a union bound and depends on two asserted properties: (i) the routing error epsilon_r is estimated separately, and (ii) per-step tool-selection error epsilon_sel(k) is monotone nondecreasing in the number of candidate tools k. The paper says this is 'a standard property of in-context classification,' but it is simply asserted, not measured for GPT-4o-mini on the 46 ERP tools, and the equality epsilon_sel(kbar) = E_i[epsilon_sel(|T_i|)] holds only when epsilon_sel is affine. Since the abstract and Section 3 present this as a 'decomposition argument linking role-aligned agents to a measurable reduction in per-step tool-selection complexity,' the authors should provide direct measurements of epsilon_sel(k) for k = 1,...,46 on their actual tool set, or at least for the relevant values, and show that the monotonicity assumption is satisfied. Otherwise Eq
  4. [§4.1 and §5.1, Tables 7–9] The task-level results are based on an LLM-as-judge that is not calibrated against human raters, and the main E1 table is a single-seed run. The stability study covers only a 10-scenario stratified subsample (n=5 runs each), and the reported confidence interval for TCR is [100%, 100%], which is not informative. The paper acknowledges these limitations in §6, but the conclusions in the abstract and Section 7 do not carry the same caveats. I ask that the main tables include the multi-seed results for all 36 scenarios, or at least that the headline TCR of 100% be replaced with a range that reflects the actual sampling variability, and that a small human-annotation study (e.g., 20–30 responses judged by an ERP practitioner) be reported to validate the automated judge.
  5. [§5.3, Table 16] The single-agent vs. multi-agent comparison is presented as evidence for the decomposition bound, but the single-agent E3 run was 'terminated at simulated day 29 because accumulated errors made further progress uninformative.' This is an uncontrolled comparison: the throughput ratio of 20x and the tool-selection error difference of 2% vs. 17% come from different experimental durations and may reflect error accumulation rather than a steady-state property. Please report the per-query tool-selection error on matched E1/E2 cells for both configurations (not on the truncated E3 run), and report the single-agent E3 metrics at day 29 (before termination) separately, so the reader can see how the comparison evolves.
minor comments (7)
  1. [Table 20] The row for 'CD1: Dispute' V2 is garbled: 'V2 653.510 13 2 2.5' should presumably read '6, 53.5, 10, 13, 2, 2.5'. Please fix the typos in this table.
  2. [Abstract and §5.4] The abstract says 'the proposed multi-agent method is significantly better than the baseline,' but in §5.4 the authors state that LangGraph does not top the ranking and that AutoGen is numerically better. The wording should be aligned with the statistical results.
  3. [§3.6] The paragraph describing tool categories ('Tools are partitioned by operational character...') is repeated verbatim. Please remove the duplicate.
  4. [Figure 7] The y-axis is labeled 'Relative Performance (%)' but the legend includes 'Stockouts (inverted)' and 'Cost Efficiency' without explaining how these are normalized or inverted. Please clarify the figure or replace it with a small table.
  5. [§5.4, Table 19] The latency comparison shows LangGraph at 21.5±8.3s, which is substantially higher than the E1 mean of 1.19s per query. The difference is explained by the crisis scenarios being more complex, but the paper should state this explicitly in the text to avoid confusion.
  6. [Table 14] The annual cost comparison excludes development, oversight, and compliance labor, as the paper notes in §6. This caveat should appear directly under Table 14 so that the 0.16% figure is not read as a total cost of ownership.
  7. [§5.1, Table 10] The routing accuracy table counts 53 queries, while the scenario suite is 36 items. The count discrepancy arises from multi-agent queries and repeated runs; please state explicitly how queries are counted.

Circularity Check

0 steps flagged

No significant circularity: the central feasibility claims rest on external simulation baselines and are not derived by construction from the paper's own definitions.

full rationale

Walking the claimed derivation chain, the decomposition bound (Eq. 4) is an analytic consequence of the explicitly stated monotonicity assumption on epsilon_sel(k), and the paper itself frames the structural arguments as motivation rather than as a substitute for measurement ('The arguments are not a substitute for measurement; they motivate the architectural choices and bound the quantities the experiments later estimate'). Predictions P1-P3 are tested through ablations (Table 21), and P3 is honestly reported as not isolated by the ablation. The E3 year-long simulation is an external benchmark against a rule-based RPA baseline on the same demand stream; nothing in the construction makes the zero-stockout result follow by definition, and the paper discloses its single-seed main-table runs, the 10-scenario stability subsample, and the LLM-as-judge proxy in Section 6. The grading rubric, acceptance criteria, and risk thresholds are author-defined, but they are applied by a separate judge and are not fitted parameters renamed as predictions; E2/E3 outcomes are not the same quantities as any input to the design. No load-bearing self-citation chain is present: the harness-engineering principles are attributed to external Anthropic sources, not to the authors' prior work. The principal weaknesses are robustness and external-validity concerns (no seed sweep, fixed RPA policy, automated judge), which the paper explicitly acknowledges; these are not circularity.

Axiom & Free-Parameter Ledger

6 free parameters · 5 axioms · 0 invented entities

The central claim rests on a large set of hand-set evaluation parameters (rubric weights, thresholds, simulation probabilities, RPA policy) and on domain assumptions about LLM error monotonicity and judge reliability. The architecture itself is a composition of known components; its contribution is the integration, which is why the ledger shows few invented entities but many tunable settings.

free parameters (6)
  • Grading rubric weights/thresholds = w=(0.30,0.30,0.25,0.15), θ=(6,7,5,5), θ̄=5.85
    Author-chosen rubric used by LLM-as-judge for all E1/E2 quality scores; no calibration against human raters.
  • Routing thresholds c_threshold and θ = not reported
    Control when multi-agent routing is triggered (Algorithm 2); trade-off precision/recall per deployment.
  • Safety harness monetary threshold = $10,000
    High-risk actions above threshold require human approval; configurable.
  • E3 simulation parameters = 10% supplier delay, 95% payment on-time, RPA (s,Q)=(50,100), demand patterns
    Hand-set stochastic model; single realization without seed variation drives the zero-stockout claim.
  • Reflector coverage threshold = 0.8
    Coverage below 0.8 triggers replan; chosen without sensitivity analysis.
  • Replan cap K = 1 (text says one replan; Table 20 shows 2 retries)
    Bounded loop cap; inconsistency between text and table.
axioms (5)
  • domain assumption Per-step tool-selection error ϵsel(k) is monotone non-decreasing in candidate-tool count k
    Invoked for bound (4) in §3.2; asserted as a standard property of in-context classification, not measured for GPT-4o-mini on these tools.
  • domain assumption LLM-as-judge scores on the author-defined rubric approximate expert assessment
    Used throughout E1/E2; §6 acknowledges it is an automated proxy not calibrated against human raters.
  • domain assumption The stochastic simulation's demand/supply models represent a realistic SME trading environment
    E3 conclusions about stockouts and cash depend on these models; only one realization is run.
  • domain assumption Odoo transactional and permission guarantees are inherited by agent tool calls
    §3.3.4: agents use XML-RPC with business-method execution and per-call transactions; assumed to ensure data integrity.
  • domain assumption Router accuracy ϵr is estimated from a labelled set and can be kept low
    Decomposition bound (4) degrades with routing error; the paper estimates it in §5.1 but does not bound it for unseen query distributions.

pith-pipeline@v1.3.0-alltime-deepseek · 28290 in / 14832 out tokens · 142143 ms · 2026-08-01T18:17:40.170985+00:00 · methodology

0 comments
read the original abstract

Enterprise Resource Planning (ERP) systems record transactions reliably but still delegate almost all operational decision-making to human specialists, because classical rule-based automation cannot reason about exceptions and monolithic AI assistants degrade when asked to coordinate across functional boundaries. This paper presents Agentic ERP, an expert-system architecture that combines role-aligned large-language-model (LLM) agents with a risk-tiered human-in-the-loop harness and a graph-based orchestrator to execute end-to-end business workflows on a production ERP backend. First, autonomous ERP operation is formulated as a constrained sequential-decision problem over a structured enterprise state, with a decomposition argument linking role-aligned agents to a measurable reduction in per-step tool-selection complexity. Second, a graph-based Planner--Executor--Reflector--Responder orchestration decouples generation from evaluation through externalised grading criteria and sprint contracts, packaging recent harness-engineering principles as inspectable expert-system artefacts. Third, the system is evaluated at three levels: a scenario-based task suite, a comprehensive comparison of six orchestration paradigms on cross-functional crisis tasks, and a 365-day agent-in-the-loop simulation against rule-based RPA and no-intervention baselines. Across these levels the proposed multi-agent method is significantly better than the baseline, and the system sustains a simulated year of operation with zero stockouts while the rule-based baseline accumulates hundreds under the same demand stream. The work shows that role-aligned LLM agents under human oversight can move an ERP system from passively recording transactions to actively executing operational decisions, and it provides a reference architecture and an evaluation protocol for autonomous enterprise resource planning.

Figures

Figures reproduced from arXiv: 2607.17331 by Lihui Wang, Tianyu Wang, Xi Vincent Wang, Zhihao Liu.

Figure 1
Figure 1. Figure 1: Agentic ERP system architecture. The four-layer design separates concerns: User Interface handles [PITH_FULL_IMAGE:figures/full_fig_p011_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Orchestration workflow. The Planner decomposes the request into a typed-tool plan; the Executor invokes [PITH_FULL_IMAGE:figures/full_fig_p012_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Online shop of the Jade Garden Tea House tea trading company. [PITH_FULL_IMAGE:figures/full_fig_p023_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Online order data in Odoo database [PITH_FULL_IMAGE:figures/full_fig_p028_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Online inventory data in Odoo database. 28 [PITH_FULL_IMAGE:figures/full_fig_p028_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: 365-day autonomous simulation results. Top: Cash position and cumulative revenue trends showing 237% [PITH_FULL_IMAGE:figures/full_fig_p030_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Baseline comparison: Multi-agent AI vs. Single-agent vs. RPA. The multi-agent system achieves highest [PITH_FULL_IMAGE:figures/full_fig_p033_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Orchestration method comparison on the 15-scenario crisis benchmark (N = 270 cells; GPT-4o-mini, three seeds, judge held constant). AutoGen and LangGraph form the top tier; the gap to ReAct is the only pairwise difference that survives Holm–Bonferroni correction at α = 0.05. See [PITH_FULL_IMAGE:figures/full_fig_p037_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Accuracy vs. latency trade-off for orchestration methods. AutoGen and the proposed LangGraph pipeline [PITH_FULL_IMAGE:figures/full_fig_p038_9.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

46 extracted references · 7 linked inside Pith

  1. [1]

    B. S. Mahapatra, D. Ghosh, D. Pamucar, G. Mahapatra, Dynamic group decision-making for enterprise resource planning selection using two-tuples pythagorean fuzzy moora approach, Expert Systems with Applications 263 (2025) 125675

    2025

  2. [2]

    URL:https://www

    Gartner, Market share analysis: ERP software, worldwide, 2024, 2025. URL:https://www. gartner.com/en/documents/6654134, published26June2025.GartnerdocumentID6654134

    arXiv 2024

  3. [3]

    URL:https://www.grandviewresearch.com/industry-analysis/ erp-software-market, last updated June 2026

    Grand View Research, ERP software market size, share & trends analysis re- port, 2026–2033, 2026. URL:https://www.grandviewresearch.com/industry-analysis/ erp-software-market, last updated June 2026

    2026

  4. [4]

    Shaul, D

    L. Shaul, D. Tauber, Critical success factors in enterprise resource planning systems: Review of the last decade, ACM Computing Surveys (CSUR) 45 (2013) 1–39

    2013

  5. [5]

    W. M. Van der Aalst, M. Bichler, A. Heinzl, Robotic process automation, Business & infor- mation systems engineering 60 (2018) 269–272

    2018

  6. [6]

    S. V. Grabski, S. A. Leech, P. J. Schmidt, A review of ERP research: A future agenda for accounting information systems, Journal of Information Systems 25 (2011) 37–78

    2011

  7. [7]

    Achiam, S

    J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Al- tenschmidt, S. Altman, S. Anadkat, et al., GPT-4 technical report, arXiv preprint arXiv:2303.08774 (2023)

    Pith/arXiv arXiv 2023

  8. [8]

    Rieger, A

    M. Rieger, A. Shah, A. Alam, J. Hossain, Possibilities and limitations of using large language models (LLMs) for alert classification and prioritisation in security operations centers (SOCs), Expert Systems with Applications (2026) 133194

    2026

  9. [9]

    Schick, J

    T. Schick, J. Dwivedi-Yu, R. Dessì, R. Raileanu, M. Lomeli, E. Hambro, L. Zettlemoyer, N. Cancedda, T. Scialom, Toolformer: Language models can teach themselves to use tools, Advances in neural information processing systems 36 (2023) 68539–68551

    2023

  10. [10]

    H. Wang, S. Zhou, Z. Chen, D. Li, J. Wang, S. Wu, S. T. Ng, Llm-driven smart agents using tools for the recovery of interdependent infrastructure networks, Expert Systems with Applications 325 (2026) 132701. 43

    2026

  11. [11]

    H. Yang, S. Yue, Y. He, Auto-GPT for online decision making: Benchmarks and additional opinions, arXiv preprint arXiv:2306.02224 (2023)

    Pith/arXiv arXiv 2023

  12. [12]

    Talebirad, A

    Y. Talebirad, A. Nadiri, Multi-agent collaboration: Harnessing the power of intelligent LLM agents, arXiv preprint arXiv:2306.03314 (2023)

    Pith/arXiv arXiv 2023

  13. [13]

    Q. Wu, G. Bansal, J. Zhang, Y. Wu, B. Li, E. Zhu, L. Jiang, X. Zhang, S. Zhang, J. Liu, et al., AutoGen: Enabling next-gen LLM applications via multi-agent conversation, arXiv preprint arXiv:2308.08155 (2023)

    Pith/arXiv arXiv 2023

  14. [14]

    S. Hong, M. Zhuge, J. Chen, X. Zheng, Y. Cheng, J. Wang, C. Zhang, S. Yau, Z. Lin, L. Zhou, et al., MetaGPT: Meta programming for a multi-agent collaborative framework, in: Interna- tional Conference on Learning Representations, volume 2024, 2024, pp. 23247–23275

    2024

  15. [15]

    C. Qian, W. Liu, H. Liu, N. Chen, Y. Dang, J. Li, C. Yang, W. Chen, Y. Su, X. Cong, et al., ChatDev: Communicative agents for software development, in: Proceedings of the 62nd annual meeting of the association for computational linguistics (volume 1: Long papers), 2024, pp. 15174–15186

    2024

  16. [16]

    Makridakis, E

    S. Makridakis, E. Spiliotis, V. Assimakopoulos, Statistical and machine learning forecasting methods: Concerns and ways forward, PloS one 13 (2018) e0194889

    2018

  17. [17]

    Kraus, S

    M. Kraus, S. Feuerriegel, A. Oztekin, Deep learning in business analytics and operations research: Models, applications and managerial implications, European Journal of Operational Research 281 (2020) 628–641

    2020

  18. [18]

    Chandola, A

    V. Chandola, A. Banerjee, V. Kumar, Anomaly detection: A survey, ACM computing surveys (CSUR) 41 (2009) 1–58

    2009

  19. [19]

    A. Xu, Z. Liu, Y. Guo, V. Sinha, R. Akkiraju, A new chatbot for customer service on social media, in: Proceedings of the 2017 CHI conference on human factors in computing systems, 2017, pp. 3506–3510

    2017

  20. [20]

    O. B. Kwon, J. Lee, A multi-agent intelligent system for efficient erp maintenance, Expert Systems with Applications 21 (2001) 191–202

    2001

  21. [21]

    P.-L. Liu, Empirical study on influence of critical success factors on erp knowledge manage- ment on management performance in high-tech industries in taiwan, Expert Systems with applications 38 (2011) 10696–10704

    2011

  22. [22]

    Arnott, G

    D. Arnott, G. Pervan, A critical analysis of decision support systems research revisited: the rise of design science, Journal of Information Technology 29 (2014) 269–293

    2014

  23. [23]

    Brown, B

    T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al., Language models are few-shot learners, Advances in neural information processing systems 33 (2020) 1877–1901

    2020

  24. [24]

    J. Wei, Y. Tay, R. Bommasani, C. Raffel, B. Zoph, S. Borgeaud, D. Yogatama, M. Bosma, D. Zhou, D. Metzler, et al., Emergent abilities of large language models, arXiv preprint arXiv:2206.07682 (2022). 44

    Pith/arXiv arXiv 2022

  25. [25]

    S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. Narasimhan, Y. Cao, ReAct: Synergizing reasoning and acting in language models, arXiv preprint arXiv:2210.03629 (2022)

    Pith/arXiv arXiv 2022

  26. [26]

    Shinn, F

    N. Shinn, F. Cassano, A. Gopinath, K. Narasimhan, S. Yao, Reflexion: Language agents with verbal reinforcement learning, Advances in neural information processing systems 36 (2023) 8634–8652

    2023

  27. [27]

    S. Yao, D. Yu, J. Zhao, I. Shafran, T. L. Griffiths, Y. Cao, K. Narasimhan, Tree of thoughts: Deliberate problem solving with large language models, Advances in Neural Information Pro- cessing Systems 36 (2023)

    2023

  28. [28]

    M. S. Sadak, A multi-agent llm framework with bayesian fusion and safety guardrails for atc- pilot communication error detection, Expert Systems with Applications 321 (2026) 132241

    2026

  29. [29]

    H. Wu, W. Zheng, M. Zhang, X. Bai, M. Pu, L. Sun, J. Liu, Y. Xu, Intention-behavior consistency-based automated failure attribution for llm-driven multi-agent systems, Expert Systems with Applications (2026) 133044

    2026

  30. [30]

    Z. Li, T. Zhou, J. Zhu, D. Li, X. Zhu, D. Wei, J. Liu, M. Song, Penexpert: A multi-agent hybrid llm–expert system framework for autonomous penetration testing, Expert Systems with Applications (2026) 133284

    2026

  31. [31]

    G. Li, H. Hammoud, H. Itani, D. Khizbullin, B. Ghanem, Camel: Communicative agents for" mind" exploration of large language model society, Advances in neural information processing systems 36 (2023) 51991–52008

    2023

  32. [32]

    Accessed: 2026-06-29

    LangChain, LangGraph: Low-level orchestration framework for building stateful agents, https://github.com/langchain-ai/langgraph, 2026. Accessed: 2026-06-29

    2026

  33. [33]

    A. A. Syntetos, Z. Babai, J. E. Boylan, S. Kolassa, K. Nikolopoulos, Supply chain forecasting: Theory, practice, their gap and the future, European journal of operational research 252 (2016) 1–26

    2016

  34. [34]

    Chu, Human–ai co-intelligence in smart smes: integrating cognitive mechanisms into ai-driven decision-making, Expert Systems with Applications (2026) 132373

    K.-M. Chu, Human–ai co-intelligence in smart smes: integrating cognitive mechanisms into ai-driven decision-making, Expert Systems with Applications (2026) 132373

    2026

  35. [35]

    H. Y. Koh, J. Ju, M. Liu, S. Pan, An empirical survey on long document summarization: Datasets, models, and metrics, ACM computing surveys 55 (2022) 1–35

    2022

  36. [36]

    Lewis, E

    P. Lewis, E. Perez, A. Piktus, F. Petroni, V. Karpukhin, N. Goyal, H. Küttler, M. Lewis, W.-t. Yih, T. Rocktäschel, et al., Retrieval-augmented generation for knowledge-intensive nlp tasks, Advances in neural information processing systems 33 (2020) 9459–9474

    2020

  37. [37]

    M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. D. O. Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, et al., Evaluating large language models trained on code, arXiv preprint arXiv:2107.03374 (2021)

    Pith/arXiv arXiv 2021

  38. [38]

    Young, Anthropic, Effective harnesses for long-running agents,https://www.anthropic

    J. Young, Anthropic, Effective harnesses for long-running agents,https://www.anthropic. com/engineering/effective-harnesses-for-long-running-agents, 2025. 45

    2025

  39. [39]

    Rajasekaran, Anthropic, Harness design for long-running application development,https: //www.anthropic.com/engineering/harness-design-long-running-apps, 2026

    P. Rajasekaran, Anthropic, Harness design for long-running application development,https: //www.anthropic.com/engineering/harness-design-long-running-apps, 2026

    2026

  40. [40]

    Mosqueira-Rey, E

    E. Mosqueira-Rey, E. Hernández-Pereira, D. Alonso-Ríos, J. Bobes-Bascarán, Á. Fernández- Leal, Human-in-the-loop machine learning: a state of the art, Artificial Intelligence Review 56 (2023) 3005–3054

    2023

  41. [41]

    L. Wang, C. Ma, X. Feng, Z. Zhang, H. Yang, J. Zhang, Z. Chen, J. Tang, X. Chen, Y. Lin, et al., A survey on large language model based autonomous agents, Frontiers of Computer Science 18 (2024) 186345

    2024

  42. [42]

    Z. Xi, W. Chen, X. Guo, W. He, Y. Ding, B. Hong, M. Zhang, J. Wang, S. Jin, E. Zhou, et al., The rise and potential of large language model based agents: A survey, Science China Information Sciences 68 (2025) 121101

    2025

  43. [43]

    Jiang, T

    Z. Jiang, T. Zhang, S. Bai, L. Lin, H. Zhang, Y. Xun, J. Ren, W. Si, S. Zhang, Towards enterprise-specific question-answering for it operations and maintenance based on retrieval- augmented generation mechanism, Expert Systems with Applications (2025) 130961

    2025

  44. [44]

    S. Wang, X. Chen, N. Han, Intelligent development of manufacturing enterprises and supply chain resilience: A perspective based on internal capabilities and external linkages, Expert Systems with Applications (2026) 131338

    2026

  45. [45]

    Wohlin, P

    C. Wohlin, P. Runeson, M. Höst, M. C. Ohlsson, B. Regnell, A. Wesslén, et al., Experimenta- tion in software engineering, volume 236, Springer, 2012

    2012

  46. [46]

    L. Wang, W. Xu, Y. Lan, Z. Hu, Y. Lan, R. K.-W. Lee, E.-P. Lim, Plan-and-solve prompting: Improving zero-shot chain-of-thought reasoning by large language models, in: Proceedings of the 61st annual meeting of the association for computational linguistics (volume 1: long papers), 2023, pp. 2609–2634. 46

    2023