{"id":"77382068-bc7c-443c-8242-eb21ed13d89b","arxiv_id":"2505.23006","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A production e-commerce chatbot using a workflow graph with node-specific prompts and response-masked fine-tuning reports large gains in accuracy and format compliance, beating a GPT-4o-based agent in human preference tests.","lead":"This paper describes a production e-commerce chatbot built around a workflow graph that routes each user turn through specialized LLM prompts and tool calls, with fine-tuning that masks responses from other graph states. The authors report large accuracy and format-compliance gains over single-prompt baselines, including against GPT-4o, in a deployed KakaoTalk service.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The headline 52% accuracy gain conflates base-model choice with the proposed method; no single baseline in Table 1 yields both stated improvements.","rationale":"The reader's CONDITIONAL verdict remains appropriate: the evaluation set is small, private, and lacks confidence intervals and human-judge correlation. My stress-test adds a distinct, more specific reporting flaw: the headline percentages in the abstract cannot be reproduced from Table 1 under a consistent baseline definition. The 52% accuracy gain is only obtained by comparing Qwen Basic to the internal fine-tuned model, while the 50% format gain is only obtained by comparing the internal Basic to the internal fine-tuned model. This mixes model-pretraining effects with the architecture and training contribution, which matters because the central claim is framed as a quantitative improvement over a baseline. The underlying architectural result is still plausible: for every base model, WG and WG-FT improve accuracy and format adherence over Basic, and the human battle test independently favors the deployed agent over GPT-4o. Therefore the paper does not need rejection, but it does need an explicit correction of the headline percentages and a clearer statement of which baseline is used for each claimed improvement. My recommended verdict is unchanged at CONDITIONAL, with this additional condition made explicit.","tokens_in":12151,"tokens_out":3632,"duration_ms":37036,"concrete_test":"Recompute each row of Table 1 as same-base-model relative improvements (B to WG and B to WG-FT) and locate which baseline produced 52% and 50%; then report the same-model percentages in the abstract and Section 4.2. If no single baseline yields both numbers, revise the abstract and the 'main contributions' bullet accordingly.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The abstract's central quantitative claim ('52% improvement in task accuracy and 50% improvement in format adherence compared to the baseline') is not supported by any single baseline in Table 1. The 52% accuracy figure matches only the comparison Qwen 2.5 32B Basic (0.578) to the internal model WG-FT (0.884): (0.884-0.578)/0.578 = 52.9%. The 50% format figure matches only the internal model Basic (0.655) to internal WG-FT (0.987): (0.987-0.655)/0.655 = 50.7%. For the same internal base model, WG-FT improves accuracy by only 19.6% (0.744 to 0.890); for the other open models the same-model gains are 42.6% (Gemma) and 52.9% (Qwen). Thus the headline mixes a cross-model gain (including the benefit of the internal model's Korean-language pretraining) with a same-model format gain, attributing both to the architecture plus response-masked fine-tuning. Because the strongest claim in the abstract is precisely this percentage improvement, the overclaim is load-bearing; the controlled evidence still shows consistent same-model gains, but the headline ratio is not reproducible from the table as stated.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This industry case study proposes a hybrid architecture for a production e-commerce conversational agent: a directed acyclic graph (DAG) workflow in which each LLM-calling node has its own system prompt, tools, history-manipulation routine, and constrained decoding, combined with response-masked fine-tuning that excludes assistant turns generated by other graph nodes. The authors compare Basic, Workflow Graph (WG), and Workflow Graph with Fine-Tuning (WG-FT) variants on Qwen 2.5 32B, Gemma 3 27B, an internal 27B-32B Korean-tuned model, and GPT-4o, using a private 161-conversation test set with LLM-as-a-judge and a coded format validator. They report that WG-FT improves accuracy by 52% and format adherence by 50% over baseline and outperforms GPT-4o, and they support this with a human battle test on the deployed KakaoTalk service.","tokens_in":12334,"tokens_out":3007,"duration_ms":29141,"significance":"If the results hold, the paper offers a practical, deployable pattern for reconciling conversational flexibility with strict business-rule compliance: decomposing a complex agent into state-specific prompts and tools, then fine-tuning with node-aware loss masking. The strengths are the real-world deployment (AI Shopping Mate), a human-preference battle test with identical external systems, and a machine-readable format validator that gives objective measurement of one key dimension. The paper also honestly acknowledges two central limitations: the human-dependent data collection process and the lack of a systematic human/LLM-judge correlation study. The main weakness is that the headline quantitative claims are not supported by a single internally consistent baseline comparison, and the small private test set without statistical inference leaves the marginal claims over GPT-4o unquantified.","major_comments":[{"comment":"The evaluation rests on 161 conversations (2,100 turns) with no confidence intervals, significance tests, or variance estimates. Several headline comparisons are very small: Internal WG-FT accuracy (0.890) exceeds GPT-4o WG (0.888) by 0.002, and the response-quality differences are around 0.02–0.06 on a 1–3 scale. With 161 conversations, these differences are within plausible sampling variability. Please report bootstrap intervals or significance tests, and clarify whether the accuracy metric is computed per turn or per conversation.","section":"Abstract and §4.2, Table 1"},{"comment":"The evaluation rests on 161 conversations (2,100 turns) with no confidence intervals, significance tests, or variance estimates. Several headline comparisons are very small: Internal WG-FT accuracy (0.890) exceeds GPT-4o WG (0.888) by 0.002, and the response-quality differences are around 0.02–0.06 on a 1–3 scale. With 161 conversations, these differences are within plausible sampling variability. Please report bootstrap intervals or significance tests, and clarify whether the accuracy metric is computed per turn or per conversation.","section":"§4.1–4.2, Table 1"},{"comment":"The test set is drawn from the same annotator-corrected prototype-agent pipeline used for training, as acknowledged in the Limitations. Because both the references and the training data share the prototype's graph-traversal outputs, the lab metrics may overstate production quality; the human battle test in §4.3 is the only evaluation on real-world traffic but covers only internal model vs GPT-4o without the Basic or WG-FT ablation. Please add an evaluation on a held-out set of real user interactions, or state this limitation in the main text together with the human-battle results.","section":"§3.2 and §6"}],"minor_comments":[{"comment":"There are typos: \"Antrophic\" should be \"Anthropic,\" and in the evaluation prompt in Figure 4, \"non-understable characters\" should be \"non-understandable characters.\"","section":"§1 and Appendix C"},{"comment":"Some table entries appear to be missing separating spaces, e.g., \"0.7900.890\" and \"0.9510.987\"; these should be rendered as distinct values in separate columns.","section":"Table 1"},{"comment":"The sentence \"After reaching final you can restart from initial\" is incomplete and should be rephrased, for example, \"After reaching the final node, the graph can restart from the initial node for multi-turn conversations.\"","section":"§3.1"},{"comment":"The metric names are inconsistent: Table 1 uses \"Response Validity\" while §4.1 calls the third dimension \"response quality.\" Unify the terminology throughout.","section":"§4.1 and Table 1"}],"recommendation":"major_revision","confidential_remarks":"This is a solid industry case study with useful engineering details, but the central quantitative claim in the abstract is overstated relative to the table, and the statistical foundation is too thin for the marginal GPT-4o comparison. The paper is likely fixable by reframing the headline around same-model gains and adding uncertainty quantification, so I recommend major revision rather than rejection. I would also flag that the private dataset and internal model make full reproducibility impossible, but that is acceptable for an applied/industry-track contribution if the claims are carefully scoped."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the response-masked fine-tuning for graph-state conversations is a real contribution, and the deployment is a genuine data point. But the abstract's 52%/50% headline is not reproducible from Table 1—the numbers come from two different baselines. That's a fixable but load-bearing problem.\n\nWhat's actually new: loss masking is old, and graph-based agents are not new, but the specific combination—training a single model across multiple DAG nodes while masking out responses generated under other nodes' system prompts—is a clean answer to a problem anyone building graph agents will hit. The paper shows consistent same-model gains from WG-FT across Qwen, Gemma, and their internal model, and the format adherence improvements are large and credible given the constrained outputs. The human battle test on the deployed KakaoTalk service is more evidence than most systems papers provide.\n\nWhere it's soft: the headline. For Qwen, accuracy rises 0.578→0.884 (52.9% relative), but format only 0.734→0.969 (32%). For the internal model, format rises 0.655→0.987 (50.7%), but accuracy only 0.744→0.890 (19.6%). No single row in Table 1 gives you both claimed percentages. The abstract should say 'up to 52%' and 'up to 50%' or pick the internal-model numbers consistently.\n\nThe evaluation is also thinner than the abstract implies: 161 conversations, no confidence intervals, no significance tests, and the primary judge is an LLM. The authors honestly flag that LLM judgments may not match human preferences, and they do provide a human battle test, but that test aggregates across categories and doesn't separate the architecture's benefit from the internal model's Korean-language pretraining. The training and test sets both come from the same annotator-corrected prototype pipeline, so the lab metrics are partly self-referential. None of this sinks the paper—for an industrial case study it's actually above average—but it means the precise numbers should be read as indicative, not definitive.\n\nBottom line: this is a useful paper for practitioners who need to enforce business constraints in conversational agents. The response-masking technique is worth stealing, and the deployed results are rare evidence. It deserves peer review, but the first request should be a rewritten abstract with honest baselines and ideally some error bars.","headline":"Solid industrial paper with a genuinely useful fine-tuning trick, but the headline 52%/50% improvements are not reproducible from Table 1 as stated.","tokens_in":12927,"tokens_out":3722,"would_cite":true,"duration_ms":33419,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A workflow graph splits a conversational agent into per-state prompts and tools, and response-masked fine-tuning lifts task accuracy 52% and format adherence 50% over a single-prompt baseline, beating GPT-4o.","keywords":["conversational agents","workflow graph","directed acyclic graph","e-commerce","fine-tuning","response masking","LLM-as-a-Judge","format adherence"],"falsifier":"Have three independent human annotators re-score the 161 test conversations with the same accuracy, format, and quality criteria, and compute agreement with the o3-mini judge; if agreement is low (for example, Cohen's kappa below 0.5 on the binary metrics), the reported 52% and 50% improvements would be artifacts of the judge rather than measurable differences in response quality.","tokens_in":11925,"feed_emoji":"🛍️","tokens_out":12226,"duration_ms":104296,"temperature":0.7,"pith_summary":"The paper tries to establish a recipe for building conversational agents that must be both flexible and strictly compliant, a tension that monolithic prompt-based LLMs resolve poorly. Its claim is that modeling the agent as a directed acyclic graph—where each node carries its own system prompt, tools, and business rules—and then fine-tuning the LLM on conversations while masking the loss of responses generated by other nodes, yields large, repeatable gains. In a private test set of 161 annotated e-commerce conversations, this approach improved task accuracy by up to 52% and format adherence by up to 50% over a single-prompt baseline across three model families, and pushed a 27B-32B internal model past GPT-4o on accuracy, format adherence, and response quality. A human preference battle test on the deployed service found the internal model preferred over the GPT-4o-based agent in safety, product recommendation, and messenger-related turns, while GPT-4o won regular chat. If the recipe works as reported, it gives production teams a practical path to controllable LLM agents without sacrificing general conversational ability.","feed_headline":"Workflow graph lifts chatbot accuracy 52%, beating GPT-4o","feed_subtitle":"Per-state prompts and masked fine-tuning raise a production shopping agent's accuracy and format adherence by about half.","key_machinery":"The load-bearing mechanism is the workflow graph, a directed acyclic graph whose nodes are either LLM calls with state-specific system prompts and custom history-manipulation routines, or external tool calls with input/output schemas; traversal of the graph replaces end-to-end generation, so each turn is governed by the short, focused instructions of its current state. The second mechanism is response-masked fine-tuning: the training sequence for a node pairs its system prompt with the full conversation, but the loss is computed only on responses that the node itself produced, so the model never learns to imitate outputs that obey a different node's constraints. The final piece is the data pipeline: a prototype agent built from the graph generates draft responses that human annotators correct, and constrained decoding forces LLM outputs that feed tool nodes to be syntactically valid JSON.","core_discovery":"The central claim is that the way an agent is structured, not just the size of its model, determines whether it can satisfy hard business constraints in production. The paper shows that replacing a single system prompt with a workflow graph—each LLM node having its own shorter prompt, few-shot examples, optional history-manipulation routine, and tool schemas—improves accuracy by up to 14% and format adherence by up to 45% relative to the monolithic baseline for the same model. Adding the proposed fine-tuning, in which the training loss is masked for any response that originated from a different graph node than the one being trained, brings further gains: the internal 27B-32B model reaches 0.890 task accuracy and 0.987 format adherence, versus 0.864 and 0.778 for the GPT-4o basic agent and 0.888 and 0.964 for the GPT-4o graph agent. The authors interpret this as evidence that graph-level decomposition plus node-aware training corrects both hallucinated tool calls and formatting violations that even state-of-the-art LLMs exhibit when given long, compound instructions.","pith_inferences":["Because the gains concentrate in format adherence and tool-call correctness, the graph is acting as a form of structured prior that reduces the effective instruction-following burden on the LLM; a testable extension would be to measure per-node prompt lengths and latency to verify the paper's implicit claim that shorter prompts are the main driver of the improvement.","The response-masking scheme is a simple, model-agnostic fix for a problem that any multi-state agent framework faces during fine-tuning, so it could plausibly be dropped into other graph-based agent frameworks that route conversations through multiple LLM nodes.","The human-vs-LLM-judge discrepancy the authors note—fluency matters to users but not to the automated judge—suggests that production evaluations should pair LLM-as-a-Judge with periodic human sampling; quantifying the human-judge agreement would be a direct follow-up.","Since the prototype agent used GPT-4o to generate draft responses before human correction, the resulting training data may carry a stylistic residue of GPT-4o; one could test whether using a weaker prototype model changes the fine-tuned agent's quality, which would bound the data pipeline's dependence on the initial generator."],"forward_implications":["Because the same workflow-graph architecture improved accuracy and format adherence for Qwen 2.5 32B, Gemma 3 27B, and the internal 27B-32B model, the recipe appears to generalize across open and proprietary model families rather than being specific to one checkpoint.","The fine-tuned graph agents let 27B-32B open-source models match or beat GPT-4o on this e-commerce workload, which implies that production teams can hit high compliance with smaller, cheaper, lower-latency models when the workflow and training are built around the graph structure.","The deployed service, live since December 2024 and covering over one million products, demonstrates that the approach survives real-world traffic and platform constraints, and the authors state the framework is adaptable to other domains with complex task execution.","The human battle test adds evidence that users prefer the graph-fine-tuned internal model over the GPT-4o-based agent in safety, recommendation, and messenger feature scenarios, with GPT-4o retaining an edge only in general chit-chat."],"supporting_citations":[{"why":"Establishes that longer system prompts degrade LLM accuracy and latency, the key motivation for keeping prompts short per graph node.","marker":"Levy et al., 2024"},{"why":"Supplies the constrained decoding technique used to make LLM outputs valid inputs for tool nodes.","marker":"Willard and Louf, 2023"},{"why":"Extends efficient structured generation for LLMs, supporting the constrained decoding applied before tool nodes.","marker":"Dong et al., 2024"},{"why":"Demonstrates LLMs can learn to call external tools from examples, the basis of the agent's tool-using nodes.","marker":"Schick et al., 2023"},{"why":"Contributes the ReAct reasoning-and-acting pattern that the workflow nodes build on.","marker":"Yao et al., 2023b"},{"why":"Shows LLMs can master many real-world APIs, supporting the multi-tool agent design used in each state.","marker":"Qin et al., 2024"},{"why":"Provides the LLM-as-a-Judge methodology used to evaluate tool-call accuracy and response quality.","marker":"Zheng et al., 2023"},{"why":"LoRA is the parameter-efficient fine-tuning method used to train the graph agent with response masking.","marker":"Hu et al., 2022"},{"why":"The Chatbot Arena battle-test format is the template for the human preference comparison against GPT-4o.","marker":"Chiang et al., 2024"}],"fun_headline_variants":["Workflow graph agent beats GPT-4o accuracy","Node-aware training lifts chatbot beyond GPT-4o","Structured prompts improve agent reliability 45%","Graph decomposition plus masked fine-tuning beats GPT-4o"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The headline gains assume that a private test set of 161 conversations, scored by an LLM judge (o3-mini) against annotator-corrected references, actually reflects what users of the deployed service experience, a concern the paper's own limitations section raises.","fun_headline_variants_meta":{"raw":{"variants":["Workflow graph agent beats GPT-4o accuracy","Node-aware training lifts chatbot beyond GPT-4o","Structured prompts improve agent reliability 45%","Graph decomposition plus masked fine-tuning beats GPT-4o"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000238,"raw_usage":{"total_tokens":1494,"prompt_tokens":911,"completion_tokens":583,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":527,"completion_tokens_details":{"reasoning_tokens":520}},"tokens_in":527,"tokens_out":583,"duration_ms":6219,"temperature":1.0,"reasoning_tokens":520,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T12:54:59.344226+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have three independent human annotators re-score the 161 test conversations with the same accuracy, format, and quality criteria, and compute agreement with the o3-mini judge; if agreement is low (for example, Cohen's kappa below 0.5 on the binary metrics), the reported 52% and 50% improvements would be artifacts of the judge rather than measurable differences in response quality.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Demonstrates LLMs can learn to call external tools from examples, the basis of the agent's tool-using nodes."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Shows LLMs can master many real-world APIs, supporting the multi-tool agent design used in each state."},{"cited_title":"Jordan, Joseph E","cited_arxiv_id":null,"evidence_quote":"The Chatbot Arena battle-test format is the template for the human preference comparison against GPT-4o."}],"review_version":1}