{"id":"7093bc96-072b-475e-ac75-ca5efbbc2b73","arxiv_id":"2503.15520","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"An LLM agent with three specialized models and execution memory follows text-defined standard operating procedures, achieving 97.8% state-decision accuracy on synthetic e-commerce seller chats with GPT-4o-mini.","lead":"This paper introduces Agent-S, a system that uses three large language models, a shared action repository, and execution memory to follow customer-care standard operating procedures step by step. In tests on three e-commerce seller support workflows, the system chose the correct next action 97.8% of the time with GPT-4o-mini on synthetic chat sessions.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 97.8% state accuracy is measured against manually labeled expected steps on synthetic sessions generated from the same SOP text the model sees; without a held-out real-world evaluation or baseline, it does not establish effective SOP navigation.","rationale":"The paper is good faith: it proposes a clean three-LLM architecture, gives full prompts, and illustrates failure handling with concrete traces. The weak point is exactly where the reader placed it: the synthetic evaluation's ground truth is not independent. The 97.8% state accuracy is a per-state label agreement on traces whose branch conditions were constructed to match the SOP text supplied to the model, so it primarily demonstrates instruction following on well-formed inputs. The live-chat claim is a single qualitative sentence with no supporting statistic, and no baseline or released data allows the reader to gauge difficulty. Because this concern directly undermines the strongest evidence for the central claim, it justifies the reader's CONDITIONAL verdict rather than full acceptance. The proposed test would settle it by measuring performance where the ground truth is independent and the task is not trivially deterministic.","tokens_in":9531,"tokens_out":5070,"duration_ms":52904,"concrete_test":"Re-run the evaluation on a held-out set of real chat transcripts (or live sessions) from the same three SOPs, with expected next actions labeled by two independent annotators blind to model outputs. Report per-session success rate, not just per-state accuracy, and compare against a deterministic baseline that selects the next action by matching the last feedback/observation to the SOP text. If the LLM does not clearly beat the baseline, or if per-session success is far below the 0.978 state accuracy, the central claim is not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central evidential claim rests on Sec. 4's state-LLM accuracy of 0.978 (Table 2), computed on 220 synthetic sessions / 1221 states. The correctness labels are assigned manually by the authors: 'If the state LLM predicts the expected step as the next action, it is labeled correct.' The expected step is derived from the same SOP workflow text that is provided verbatim to the state LLM in the prompt (Appendix A.1), and the synthetic observations/API responses are generated so that SOP branch conditions (e.g., 'if its inactive,' 'if API call failed') are satisfied. This creates a closed loop: the test measures whether GPT-4o-mini can follow a cleanly written deterministic flowchart with well-formed memory entries, not whether the agent navigates realistic, ambiguous, or previously unseen situations. The live-chat evaluation is reported only as 'good accuracy' with no numbers, error analysis, or inter-annotator agreement, so it cannot independently validate the synthetic result. A simple rule-based baseline that maps (last observation, feedback) to the next line of the SOP would likely achieve high accuracy on these synthetic traces; without such a baseline, 0.978 is not evidence of agentic reasoning or fault tolerance beyond text following.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Agent-S, an LLM-based agentic workflow for automating Standard Operating Procedures (SOPs) in customer-care operations. The architecture uses three task-specific LLMs (state decision, action execution, user interaction), a Global Action Repository (GAR), textual execution memory with action/observation/feedback entries, and external environments (API tools, user interface, RAG-based knowledge source). The SOP is supplied as a plain-text logical block with indentation. The system is evaluated on three e-commerce seller SOPs using 220 synthetic chat sessions (1,221 states) and an unspecified set of in-house live chat sessions. The state LLM achieved 0.978 accuracy with GPT-4o-mini versus 0.565 with GPT-3.5, and per-task action LLM accuracies are reported for question generation, parameter extraction, and search query generation. The authors conclude that the agent can effectively navigate SOP flows and is scheduled to go live after engineering integration.","tokens_in":9761,"tokens_out":3265,"duration_ms":35184,"significance":"The contribution is an applied system design with concrete engineering choices: a shared global action repository, a single state-decision prompt across SOPs, and a fault-tolerant mechanism that repeats dependent steps or seeks external knowledge on failure. The paper includes all LLM prompts in the appendix, which is a useful reproducibility asset. If the evaluation were strengthened, the work would provide practical evidence that plain-text SOPs can be automated by a compact three-LLM agent. However, the current evidence is not yet convincing for the central claim: the synthetic evaluation is a self-consistency check on traces generated from the same SOP text that the agent is prompted to follow, the live evaluation is not quantified, and no baseline or ablation isolates the contribution of the LLM reasoning. The authors are transparent about the manual labeling procedure and about conditioning action LLM accuracy on correct state decisions, which is commendable but does not remove the need for a stronger evaluation.","major_comments":[{"comment":"The 0.978 state accuracy is measured on synthetic sessions generated from the same three SOP texts that are provided verbatim to the state LLM in the prompt (Appendix A.1), and the 'expected step' is manually assigned using those same texts. This is a closed-loop self-consistency check: it tests whether GPT-4o-mini can follow a cleanly written flowchart when branch conditions in the synthetic API responses and user inputs are sampled to satisfy the SOP logic. It does not measure navigation of ambiguous, contradictory, or previously unseen real-world situations. To support the claim of effective SOP navigation, please evaluate on held-out real or realistically noisy sessions with independent ground truth, or at minimum report inter-annotator agreement on the manual labels and compare the synthetic input/response distribution with actual chat logs.","section":"Section 4, Table 2"},{"comment":"No baseline or ablation is reported. A deterministic controller that reads the current execution memory and the SOP text and outputs the next line according to the last feedback and observation would likely achieve high accuracy on these synthetic traces, because the data generation ensures the observations satisfy branch conditions (e.g., 'if its inactive,' 'if API call failed'). Without a baseline, the 0.978 result does not demonstrate that the LLM's logical reasoning—rather than simple text following—is responsible for the performance. Please include a rule-based baseline or an ablation that removes or degrades the execution memory, and show that the LLM agent significantly outperforms it.","section":"Section 4, Table 2"},{"comment":"The action LLM accuracy is evaluated 'only for the cases where the state LLM decision is correct,' and parameter extraction accuracy is explicitly dependent on the user LLM's slot extraction accuracy. These are conditional metrics, not end-to-end pipeline accuracies. A single state-LLM error can invalidate an entire session, and the reported component accuracies do not quantify overall task success. Please report an end-to-end session success rate (e.g., fraction of sessions that reach the correct termination without any erroneous step), or provide an error decomposition that shows what the unconditional accuracy would be.","section":"Section 4, Table 3"},{"comment":"The phrase 'We observed good accuracy for the session success' is the only description of the live chat evaluation. Since the synthetic evaluation is a self-consistency check, the live chat results are the only potentially independent evidence for the claim of real-world effectiveness. Please provide the number of live sessions, the definition of session success, the quantitative success/accuracy rate, a breakdown of failure modes, and the annotation procedure (including any inter-annotator agreement). Without these details, the live evaluation cannot be independently assessed.","section":"Section 4, live chat evaluation"}],"minor_comments":[{"comment":"The prompt texts contain a JSON formatting typo: 'Output JSON should have keys \"thought, \"next_action\"' is missing the closing quote and bracket. It should be '\"thought\", \"next_action\"' and the JSON example should be closed properly. Since the appendix is provided for replicability, this should be corrected.","section":"Appendix A.1, A.2, A.3"},{"comment":"The column header 'gpt4o-mini' should be written as 'gpt-4o-mini' for consistency with Table 2 and the text.","section":"Table 3"},{"comment":"The phrase 'all possible API responses' overstates the coverage: the set of API responses was constructed by the authors and is finite. Please clarify the sampling distribution and state which responses (success, invalid ID, API failure, etc.) were included and how many variants were used.","section":"Section 4"},{"comment":"The action repetition limit (more than two repetitions triggers termination with a grace message) is a free parameter with no sensitivity analysis. Reporting how the success rate and user experience vary with this limit would strengthen the design claims.","section":"Section 3"},{"comment":"The paper describes the SOP as a Directed Acyclic Graph, yet the agent explicitly repeats actions on failure, which creates cycles in the execution trace. Please clarify whether the DAG statement applies only to the SOP definition or also to the agent's execution semantics.","section":"Section 1 and Section 3"}],"recommendation":"major_revision","confidential_remarks":"The manuscript presents a practical system with a plausible architecture and useful transparency (full prompts, manual evaluation criteria). However, the central claim of effective SOP navigation rests on a circular synthetic evaluation and an unquantified live-chat statement. The requested baselines, end-to-end metrics, and quantitative live results are within the scope of a revision; there is no indication of a fundamentally unsound architecture. I therefore recommend major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The architecture is the most concrete part of this paper, and it's worth a look. Instead of a state machine with a separate prompt per state, the author uses one state-decision prompt that reads the SOP written as an indented text block plus the execution memory, and picks the next action from a global repository. The three-LLM split (state, action, user) is simple, and the retrieval-based mapping from the state LLM's free-form output to canonical actions is a nice touch. Full prompts are in the appendix, which is a real plus for replicability. The walkthrough tables showing what happens on API failure, invalid ID, and user questions give a concrete sense of the system's fault handling.\n\nThe soft spot is the evaluation. The 0.978 state accuracy is measured on synthetic sessions generated from the same SOP text the model is given, and the expected-step labels are assigned manually against that same text. That makes it a self-consistency check, not a measure of real-world navigation. There's no baseline—a deterministic next-line / repeat-on-failure procedure might do surprisingly well on these traces. The action LLM accuracy is computed only after correct state decisions, which makes the end-to-end picture look better than it is. And the live chat evaluation is one sentence: 'good accuracy,' with no numbers, no error analysis, no inter-annotator agreement.\n\nThose are real limitations, but for an applied industry paper they're not disqualifying. The architecture is coherent, the prompts are honest, and the 0.978 is not some fabricated number—it just doesn't license the abstract's claim about 'complex real-world scenarios.' The author would have a stronger paper by framing the synthetic result as what it is and reporting the live trial properly.\n\nCitation pattern is fine. The related work is brief but points to Stateflow and the memory survey, and no relevant prior work is ignored. No code or data released, but prompts plus three SOPs give a reasonable starting point.\n\nWho is this for? Practitioners building LLM agents on deterministic workflows, and anyone who wants a concrete example of prompt design and error handling. I'd send it to a serious referee who can push on evaluation design and baselines; with heavy revision it can become a reasonable applied contribution. I would not desk-reject it.","headline":"A workable three-LLM architecture for SOP automation with honest prompts, but the 97.8% headline number is an in-sample self-consistency score, not a real-world validation.","tokens_in":10258,"tokens_out":3007,"would_cite":true,"duration_ms":29989,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Three task-specific LLMs automate Standard Operating Procedures, and with GPT-4o-mini the agent picks the expected next action 97.8% of the time.","keywords":["LLM agents","Standard Operating Procedure","customer care automation","state decision LLM","execution memory","Global Action Repository","e-commerce seller support","retrieval augmented generation"],"falsifier":"Take a held-out set of real historical customer-care transcripts for the same three SOPs, feed each user turn and each actual API response to the agent, and compare the agent's predicted next action against the action the human agent took next. If accuracy on those real transcripts falls substantially below the 0.978 synthetic figure, the synthetic evaluation did not capture production conditions and the claim that the agent effectively navigates real SOPs would not be supported.","tokens_in":9324,"feed_emoji":"🤖","tokens_out":9007,"duration_ms":88577,"temperature":0.7,"pith_summary":"The paper tries to establish that a Standard Operating Procedure can be automated by an LLM agent that treats every step as either a user interaction or a status check (API call), with a single state-decision LLM choosing each next action from the SOP text plus an execution memory. This matters because SOPs in customer care are usually written documents for humans, and this design would let them become runnable automated flows without retraining a model or imposing a formal workflow schema. The reported evidence is that ChatGPT-4o-mini as the state LLM selected the expected next action 97.8% of the time across 1221 states drawn from synthetic sessions over three e-commerce seller SOPs, and that the agent recovers from failures by repeating the failed step or an earlier dependent step. A sympathetic reading is that this is a practical architecture for converting existing written procedures into LLM-driven workflows.","feed_headline":"LLM agent picks the right SOP step 97.8% of the time","feed_subtitle":"A single decision model plus action and user models turns written procedures into automated customer care.","key_machinery":"The load-bearing mechanism is the pairing of a text-based SOP workflow with a textual execution memory. The workflow is not a formal schema; it is a natural-language block with indentation for sub-flows (e.g., 'if its active or on-hold: ask user to provide listing id'). The state-decision LLM consumes that block together with the execution memory, whose entries are triples of action, observation, and feedback, and outputs the next action. A retrieval model encodes the output and matches it by cosine similarity to an action in the Global Action Repository, which supplies the action type, required parameters, and API endpoint. The action-execution LLM then talks to the appropriate environment, the resulting observation and success or fail feedback are appended to memory, and the loop repeats. This design lets failure handling be reasoned about rather than pre-scripted: invalid IDs trigger a re-ask, API failures trigger a retry, and user questions trigger a branch to external knowledge followed by resumption of the flow.","core_discovery":"The paper's central claim is that SOP navigation is a language task: if the SOP is written as a plain logical block of text and the system remembers every action, observation, and feedback triple, then a sufficiently strong LLM can decide the next action, and two supporting LLMs can execute it. The specific architecture has three task-specific LLMs: a state-decision LLM that outputs the next action using chain-of-thought, an action-execution LLM that generates user questions, API parameters, messages, or knowledge-base search queries, and a user-interaction LLM that validates inputs, extracts slots, spell-corrects, and acknowledges. A Global Action Repository stores all possible actions and their metadata, and an embedding model matches the state LLM's free-text output to the closest repository action. On the three SOPs, GPT-4o-mini achieved 0.978 state accuracy versus 0.565 for GPT-3.5, while action execution for question generation and parameter extraction was near 1.0 and search-query generation reached 0.951; the paper interprets this as demonstration that an LLM with good reasoning can effectively run the SOP flow, including fault recovery.","pith_inferences":["The paper reports live-chat results only as 'good accuracy' and does not give a live per-state number; a direct extension would be to report live state accuracy with the same metric used for synthetic sessions so the 0.978 figure can be compared under real input distributions.","The state LLM's choice of which previous action to repeat is effectively a learned repair policy; comparing those choices with the recovery moves human agents make in historical transcripts would show how closely the agent mirrors human judgment.","Because the Global Action Repository and state prompt are SOP-agnostic, a cheap test of domain-independence would be to apply the architecture to an SOP from another industry using only a new workflow text and new repository rows.","The current loop-prevention rule stops after two repeats of the same action; varying this threshold per action type and measuring resolution rates would tell whether the same constant is safe for API retries and user-input re-asks."],"forward_implications":["A new SOP can be onboarded by writing its workflow as a plain-text logical block and adding its actions to the shared Global Action Repository, with no new prompts or per-state models.","The agent is designed to handle failures by repeating a failed API call, re-asking for invalid user input, or returning to an earlier dependent step, and it terminates gracefully after two repeated attempts.","User questions raised mid-flow are answered by branching to an external knowledge source and then resuming the same SOP step, so the conversation need not restart.","Because the SOP is just text and the action repository is shared across procedures, the same state-decision prompt should carry over to any workflow representable as a DAG, not just the three e-commerce seller SOPs tested."],"supporting_citations":[{"why":"Supplies the e5-base-v2 embedding model that matches the state LLM's free-text next-action output to a Global Action Repository action.","marker":"Wang et al., 2022"},{"why":"Defines the prior LLM state-machine paradigm that the paper contrasts with; the single state prompt across all SOPs is the claimed flexibility advantage.","marker":"Wu et al., 2024"},{"why":"Provides the LLM-agent web-navigation precedent that motivates letting the agent interact with user and API environments.","marker":"Abuelsaad et al., 2024"}],"fun_headline_variants":["LLM agent automates SOPs with 97.8% step accuracy","Agent-S: LLM trio navigates standard operating procedures","Memory-augmented LLM agents execute SOPs fault-tolerantly","Three LLMs, one workflow: automating SOPs with Agent-S","Agent-S: LLM reads SOP and acts, 97.8% correct"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The evaluation assumes that synthetic sessions built from randomly sampled user inputs and API responses represent real customer-care interactions closely enough that the 97.8% state accuracy on them predicts how the agent will perform with live users and production system behavior.","fun_headline_variants_meta":{"raw":{"variants":["LLM agent automates SOPs with 97.8% step accuracy","Agent-S: LLM trio navigates standard operating procedures","Memory-augmented LLM agents execute SOPs fault-tolerantly","Three LLMs, one workflow: automating SOPs with Agent-S","Agent-S: LLM reads SOP and acts, 97.8% correct"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000767,"raw_usage":{"total_tokens":3440,"prompt_tokens":1022,"completion_tokens":2418,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":638,"completion_tokens_details":{"reasoning_tokens":2323}},"tokens_in":638,"tokens_out":2418,"duration_ms":20248,"temperature":1.0,"reasoning_tokens":2323,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-09T16:16:27.975035+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a held-out set of real historical customer-care transcripts for the same three SOPs, feed each user turn and each actual API response to the agent, and compare the agent's predicted next action against the action the human agent took next. If accuracy on those real transcripts falls substantially below the 0.978 synthetic figure, the synthetic evaluation did not capture production conditions and the claim that the agent effectively navigates real SOPs would not be supported.","supporting_citations":[],"review_version":1}