{"id":"d2950731-6262-4a81-83e7-b037419704ac","arxiv_id":"2505.05445","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"clem:todd is a self-play benchmarking framework for task-oriented dialogue systems, and its MultiWOZ 2.2 booking experiments show monolithic and LLM-managed modular systems beating a fixed pipeline at lower cost.","lead":"This paper introduces clem:todd, an evaluation framework built on the authors' clembench self-play platform, in which one LLM plays the user and another plays the booking assistant, with a game master enforcing a JSON tool schema. It reports task success, cost, and latency for three new dialogue system designs and two prior systems on a filtered subset of MultiWOZ 2.2 (117 booking tasks).","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Headline architecture rankings are conditioned on one user simulator; Table 2 shows simulator choice changes task success by up to 0.58, so Tables 3/4/Figure 3 may not generalize to real users.","rationale":"The paper's own Table 2 is the strongest evidence for the concern: it was designed to quantify sensitivity to simulator choice, yet the main experiment design then collapses that variability by selecting one simulator and interpreting Table 3 as if it measured dialogue systems alone. For a benchmarking framework whose promise is consistent conditions, fixing one simulator is defensible as a controlled condition, but the 'actionable insights' in Figure 3 generalize only if the chosen simulator is representative of real user behavior. The supporting Turing test has a single annotator and 100 comparisons, with no inter-annotator agreement or significance testing reported. I agree with the reader's weakest assumption. I do not recommend rejection: the framework contribution and open-source implementation are real, and the concern can be settled by re-running with a second simulator and reporting uncertainty. Since the reader already conditions acceptance on similar fixes, the verdict stays unchanged. The secondary internal inconsistencies (US-spread defined as range but captioned as standard deviation in Table 2; repeated MultiWOZ cells in Table 13) should also be corrected, but they are not the load-bearing issue.","tokens_in":28812,"tokens_out":6120,"duration_ms":65544,"concrete_test":"Re-run the Table 3 / Figure 3 comparisons (all architectures and the 117 booking tasks) with a second user simulator, e.g., Llama-3.3-70B with the same prompt or a rule-based ConvLab2 user, and recompute booking accuracy and FLOP/token cost. If the architecture ranking changes, or if any booking accuracy shifts by more than about 0.10 between simulators, report the headline trade-offs as simulator-dependent and add error bars via a paired bootstrap over the 117 tasks.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 5.1.2 fixes Qwen2.5-32B as the only user simulator for all subsequent comparisons, but Table 2 shows that the same dialogue system's booking accuracy swings by up to 0.58 (Qwen2.5-32B Monolithic: 0.42 with Llama-3.2-1B as user vs 1.00 with Llama-3.3-70B as user) depending on which LLM plays the user. The selection of Qwen2.5-32B rests on LLM-judge naturalness scores and a single-annotator Turing test over 100 dialogues (Appendix A.4), not on evidence that this simulator preserves the relative ordering that real users would produce. Because the user simulator adapts to the system and decides when to emit DONE, every Table 3/Figure 3 booking-accuracy value is a property of the (system, simulator) pair, not of the system alone. With only 117 tasks and no confidence intervals, small headline gaps such as Monolithic 0.71 vs Modular-LLM 0.68 are within sampling noise, while the larger gap to Modular-Prog 0.41 could flip if a different simulator is less forgiving of format violations or more likely to terminate early. The framework itself is a useful infrastructure contribution, but the architecture trade-offs are not yet established as simulator-independent.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes clem:todd, a framework for benchmarking LLM-based task-oriented dialogue systems in a self-play setup derived from clembench. A user simulator (Player A) and a dialogue system (Player B) interact through a Game Master that enforces a JSON tool schema and executes database lookups and booking validations. The framework is instantiated on a filtered MultiWOZ 2.2 test set of 117 booking tasks, using Inform and Booking Accuracy plus cost and latency metrics. The authors integrate two existing systems (AutoTOD and the Hudecek and Dusek modular pipeline) and introduce three architectures (Monolithic, Modular-Prog, Modular-LLM), evaluating them across Llama, Qwen, and GPT-4o models. They report that monolithic systems and Modular-LLM achieve higher booking accuracy at lower or comparable cost than Modular-Prog, that small models fail mainly through format violations, and that the framework can be adapted to synthetic MultiWOZ-style and unrealistic datasets.","tokens_in":29084,"tokens_out":5192,"duration_ms":55105,"significance":"The framework contribution is genuinely useful: it provides a unified, reproducible evaluation harness with released code, prompt templates, tool schemas, and a consistent cost model. The evaluation is anchored to an external benchmark with standard metrics, and the paper does not fit any quantity to a target result, so circularity is not a concern. If the empirical findings are robust, the comparison of monolithic versus modular control strategies and the analysis of format-violation failures would be actionable for practitioners. However, the headline architecture rankings are heavily conditioned on the choice of a single user simulator, the test set is small, and at least one appendix table is internally inconsistent. These issues make the current empirical conclusions conditional rather than established.","major_comments":[{"comment":"The decision to fix Qwen2.5-32B as the user simulator for all subsequent experiments is not sufficiently justified. Table 2 shows that the same dialogue system's booking accuracy varies by up to 0.58 depending on the simulator model (e.g., Qwen2.5-32B Monolithic: 0.42 with Llama-3.2-1B versus 1.00 with Llama-3.3-70B). The selection argument in Section 5.1.2 is based on naturalness/coherence scores and a Turing test, but there is no evidence that this simulator preserves the relative ordering of architectures that another simulator, or real users, would produce. Consequently, every value in Tables 3, 4, and 6 and Figure 3 is a property of the (system, simulator) pair, not of the system alone. Please either report results across the full panel of simulators, or provide rank-stability evidence showing that the architecture ordering is invariant to simulator choice, or substantially weaken the architecture-level conclusions.","section":"Section 5.1.2, Tables 2 and 3"},{"comment":"The caption of Table 2 states that the bottom row reports the 'standard deviation across user simulators', but Section 5.1.2 defines US-spread as the range (maximum minus minimum) of task success rates. The arithmetic in the table matches the range definition (e.g., 1.00 - 0.42 = 0.58 for Qwen2.5-32B Monolithic), not a standard deviation. This is an internal inconsistency in a metric that is central to the robustness analysis; please correct the caption or the definition and ensure the reported values match the stated statistic.","section":"Table 2 caption and Section 5.1.2"},{"comment":"The Multi-domain Modular-Prog rows in Table 13 contain identical values (0.80, 0.50, 0.45, 0.58) for four different models: Llama-3.2-1B, Llama-3.2-3B, Qwen2.5-7B, and Llama-3.1-8B. This pattern strongly suggests a copy-and-paste error. Moreover, these entries are inconsistent with Table 3: for example, Llama-3.2-1B has a Modular-Prog booking accuracy of 0.00 in Table 3, yet the single-domain and multi-domain averages in Table 13 would imply a weighted average of approximately 0.28, not 0.00; similarly, Qwen2.5-7B would imply approximately 0.52 rather than the reported 0.26. Please correct Table 13 and re-verify every appendix table against the headline results, since this undermines confidence in the domain-level analysis.","section":"Appendix A.5, Table 13"},{"comment":"Headline comparisons such as Monolithic 0.71 versus Modular-LLM 0.68 are presented without confidence intervals or significance tests on the 117-task filtered set. With n = 117, the binomial standard error at p = 0.71 is about 0.042, so a difference of 0.03 is well within sampling noise. The larger gap to Modular-Prog (0.41) is meaningful, but the paper's claim that Modular-LLM achieves 'comparable performance' to Monolithic is not statistically supported. Please add confidence intervals or explicit uncertainty estimates, and adjust the wording so that the descriptive differences are not presented as established rankings.","section":"Section 5.1.3 and Figure 3"},{"comment":"The monolithic dialogue system prompt in Figure 7 appears to contain instructions copied from the user simulator template: it says 'Communicate naturally by expressing preferences, asking clarifying questions, and making decisions as needed' and instructs the system to 'reply with DONE' and to 'not simulate or act as the dialogue system'. If this prompt was actually used, it is inappropriate for a dialogue system and would undermine the monolith results; if it is an appendix error, the template must be corrected because the paper presents the appendix as the exact prompt used. Please clarify and fix this discrepancy.","section":"Appendix A.2, Figure 7 (Prompt Template for Monolithic System)"}],"minor_comments":[{"comment":"The filtering to 117 booking-only tasks is described briefly; please add a sentence on how this selection may affect generalizability to non-booking MultiWOZ dialogues and why the 1000-task test split was not used directly.","section":"Section 4"},{"comment":"The row label 'User Spread' should be made consistent with the term 'US-spread' used in Section 5.1.2 to avoid confusion.","section":"Table 2"},{"comment":"The x-axis label reads 'FLOPs per token' while the surrounding text and Table 4 describe FLOP cost per dialogue; please make the units consistent.","section":"Figure 3"},{"comment":"The cost estimates depend on the assumption of $0.05 per petaFLOP and on OpenRouter prices at the time of writing; a short sensitivity discussion would be helpful since these assumptions directly drive the cost comparisons in Table 4 and Figure 3.","section":"Appendix A.6"},{"comment":"The sentence 'individual modules operate without access to the full dialogue history' is presented as an explanation for modular failures; please make clear whether this is an observed property of the implementation or an interpretive claim, since the prompts in Figures 10-12 appear to include dialogue history in the input.","section":"Section 5.1.3"}],"recommendation":"major_revision","confidential_remarks":"The framework and released resources are a solid infrastructure contribution, and the paper is likely a good fit for the journal. My main concern is that the authors' own Table 2 directly undermines the single-simulator design: with task success varying by up to 0.58 across simulators, the architecture rankings in Tables 3 and 4 cannot be taken as simulator-independent. I would require either multi-simulator reporting or explicit rank-stability evidence before acceptance. The Table 13 inconsistency and the Figure 7 prompt-template discrepancy are concrete errors that should be fixed in revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First thing to know: this is a genuinely useful evaluation harness, not a settled set of architecture rankings. The framework runs, the code is out, and the self-play setup with a JSON tool schema is a sensible adaptation of clembench to task-oriented dialogue. The second thing: the headline comparisons in Tables 3/4/Figure 3 are conditioned on a single user simulator, and their own Table 2 shows that swapping the simulator changes booking accuracy for the same dialogue system by up to 0.58. That makes the 'actionable insights' about architecture less robust than the prose suggests.\n\nWhat is actually new: clem:todd extends clembench with tool-call constraints, booking metrics, cost/latency tracking, and synthetic robustness sets. Re-evaluating AutoTOD and Hudecek & Dusek under one harness is useful, and the appendix is transparent about integration patches (regex fixes, case-insensitive columns). The cost/flop analysis is a nice addition even though prices are time-sensitive.\n\nSoft spots, in proportion:\n\nThe user-simulator dependence is the load-bearing one. They pick Qwen2.5-32B for all subsequent experiments because of LLM-judge naturalness and a single-annotator Turing test over 100 dialogues. No evidence that this simulator preserves the ordering real users would produce. They even note that Qwen-based systems are more robust to weak simulators, which means the choice of simulator can favor certain architectures. This should be a headline limitation, not a footnote.\n\nThe statistics are thin: 117 tasks, no confidence intervals. The gap between Monolithic 0.71 and Modular-LLM 0.68 is noise. Even the 0.71 vs 0.41 gap could flip if a different simulator is less forgiving of format violations. They do provide domain-level breakdowns, which helps, but no error bars.\n\nMinor issues: US-spread is described as a range but captioned as standard deviation; Table 13 has a visible copy-paste error (several multi-domain Modular-Prog rows identical at 0.58). The 'first plug-and-play framework' claim is overstated given they cite ConvLab2 and ParlAI. These are fixable.\n\nOverall: solid infrastructure paper that deserves referee time. The framework will be useful to anyone benchmarking LLM-based TOD systems, and the robustness analysis (when cleaned up) will be a good methodological contribution. I'd accept for review with a request for major revision on the simulator-dependence framing and statistical reporting.","headline":"Useful TOD benchmarking harness, but its architecture rankings are simulator-dependent; worth engaging as infrastructure, not as settled results.","tokens_in":29615,"tokens_out":2500,"would_cite":true,"duration_ms":26756,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"One self-play loop can fairly benchmark any user-simulator and dialogue-system pairing, and its results favour monolithic and LLM-modular designs over a fixed programmatic pipeline.","keywords":["task-oriented dialogue","LLM user simulation","self-play benchmarking","clem:todd","tool call schema","booking accuracy","MultiWOZ","dialogue system evaluation"],"falsifier":"Take the same 117 booking tasks and the same three architectures, replace the LLM user simulator with transcripts from human users (or with a differently built simulator that does not defer to the system's corrections), and recompute booking accuracy. If the ranking of Monolithic, Modular-LLM, and Modular-Prog changes, or if GPT-4o's advantage over open models shrinks, the framework's architecture trade-off was driven by simulator behaviour rather than by the systems themselves.","tokens_in":28606,"feed_emoji":"🎮","tokens_out":8434,"duration_ms":81756,"temperature":0.7,"pith_summary":"Task-oriented dialogue systems are usually tested in isolation, so it is hard to say whether one architecture beats another because of the system or because of the test rig. This paper claims that clem:todd, a self-play benchmark built around an LLM 'user' conversing with a dialogue system under a strict JSON tool-call schema, gives a unified answer: the same dataset, metrics, and compute budget apply to any simulator–system pairing. Under that rig, the framework's experiments conclude that monolithic systems and LLM-controlled modular systems achieve higher booking accuracy than a fixed programmatic modular pipeline, at lower or comparable cost, and that the largest cause of small-model failure is format violation rather than dialogue understanding. The paper also establishes that swapping the user simulator can change a system's success rate by up to 0.58, so single-simulator scores should be read as conditional on the simulator.","feed_headline":"Booking chatbots: fixed pipelines lose to monolithic and LLM-modular","feed_subtitle":"A self-play benchmark shows architecture choices matter, and simulated-user choice can flip the results.","key_machinery":"The load-bearing mechanism is clem:todd itself: a game-master-coordinated two-player loop in which an LLM user simulator, an LLM dialogue system, a schema-constrained tool-call interface, and automatic scoring are bound into one reproducible pipeline. The Tool Schema—a fixed set of JSON function calls such as retrievefromrestaurantdb and validatetrainbooking—is what makes the comparison clean: every system must express its behaviour as the same machine-executable actions, and any format violation ends the episode. This is the object that carries the argument, because it converts 'which architecture is better' into a question answerable under identical data, metrics, and compute for every pairing.","core_discovery":"The discovery this paper advances is that an evaluation framework for task-oriented dialogue can be built by treating a booking task as a two-player assistance game, and that doing so makes architecture trade-offs visible and comparable. Concretely, a user simulator with a natural-language goal talks to the dialogue system; the dialogue system must reply with structured tool calls (query database, validate booking, follow up); a game master enforces the schema, aborts on any violation, executes the calls, and scores the final booking against ground truth. The paper claims this is the first setup to combine plug-and-play integration of different simulators and systems with uniform datasets, metrics, and compute constraints. Used on the MultiWOZ 2.2 booking tasks, it finds that Monolithic and Modular-LLM realisations outperform Modular-Prog on booking accuracy and cost, and that small models are defeated mainly by response-format violations rather than by dialogue failure.","pith_inferences":["If format violations are the dominant small-model failure, then adding constrained decoding or grammar-restricted sampling at the tool-call layer is a natural next test that clem:todd's schema makes easy; the paper does not run it, but its own data predicts a large gain for 1B–8B models.","The reported simulator sensitivity suggests that many published single-simulator TOD rankings may be confounded; rerunning prior systems under clem:todd with a second simulator would quantify how much of the literature's ordering is rig-dependent.","The unrealistic-task results imply that correcting a user's impossible request counts as a failure under strict goal matching; a separate 'goal-aware repair' metric would distinguish systems that sensibly steer the user from systems that simply lost the goal, which the paper does not compute.","The Turing test for simulator naturalness rested on one annotator; a multi-annotator replication with agreement statistics would be the minimal check on whether Qwen2.5-32B is genuinely more human-like, and hence on the simulator choice itself."],"forward_implications":["Architecture rankings become commensurable: a system's score in clem:todd can be compared across papers because data, metrics, and compute are fixed by the rig rather than by each lab's choices.","Practitioners can pick by constraint: where compute or API cost dominates, monolithic systems are the cheaper route to high booking accuracy; where interpretability matters, Modular-LLM buys part of the benefit without Modular-Prog's cost.","Small-model results should be read as an instruction-following problem first: most small-model failures are aborted dialogues caused by invalid tool-call formats, so better schema adherence could raise their booking accuracy substantially.","Evaluators should report robustness to simulator choice rather than a single number: the observed user-spread of 0.25–0.62 means a system's reported success depends on the simulated user, and the paper recommends testing across several.","The rig can move to new goals: because the game logic is separate from the test instances, synthetic MultiWOZ-style and even deliberately unrealistic tasks can be injected without changing the evaluation logic."],"supporting_citations":[{"why":"Supplies the two-player game-master self-play loop that clem:todd adapts to task-oriented dialogue.","marker":"(Chalamalasetti et al., 2023)"},{"why":"Provides the MultiWOZ 2.2 goals, dialogues, and ground-truth annotations used for all task-success scoring.","marker":"(Budzianowski et al., 2018)"},{"why":"Contributes the AutoTOD baseline and the booking-accuracy metric adopted as the primary task-success measure.","marker":"(Xu et al., 2024)"},{"why":"Contributes the modular-pipeline baseline system that is re-evaluated inside the framework.","marker":"(Hudecek and Dusek, 2023)"},{"why":"Provides the LLM-as-a-judge protocol for naturalness, coherence, and diversity of simulator utterances.","marker":"(Kazi et al., 2024)"},{"why":"Grounds the FLOP estimation of 2× parameters per token used in the cost comparisons.","marker":"(Kaplan et al., 2020; Chowdhery et al., 2023)"}],"fun_headline_variants":["Booking chatbot benchmark: simulator choice can flip results","Monolithic and LLM-modular beat modular-prog in bookings","New benchmark exposes architecture trade-offs in dialogue","Self-play game evaluates dialogue systems consistently"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The headline rankings stand on the choice of Qwen2.5-32B as the user simulator for all the comparisons; if that simulated user is more accommodating, predictable, or goal-faithful than real customers, the winner board is an artifact of the simulator. The paper's Limitations section also flags that its modular pipelines are deliberately simple and that strict format enforcement aborts many small-model dialogues, so the scores are conditional on those design choices.","fun_headline_variants_meta":{"raw":{"variants":["Booking chatbot benchmark: simulator choice can flip results","Monolithic and LLM-modular beat modular-prog in bookings","New benchmark exposes architecture trade-offs in dialogue","Self-play game evaluates dialogue systems consistently"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000208,"raw_usage":{"total_tokens":1391,"prompt_tokens":918,"completion_tokens":473,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":534,"completion_tokens_details":{"reasoning_tokens":413}},"tokens_in":534,"tokens_out":473,"duration_ms":5072,"temperature":1.0,"reasoning_tokens":413,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T23:03:21.679013+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the same 117 booking tasks and the same three architectures, replace the LLM user simulator with transcripts from human users (or with a differently built simulator that does not defer to the system's corrections), and recompute booking accuracy. If the ranking of Monolithic, Modular-LLM, and Modular-Prog changes, or if GPT-4o's advantage over open models shrinks, the framework's architecture trade-off was driven by simulator behaviour rather than by the systems themselves.","supporting_citations":[],"review_version":1}