Pith. sign in

REVIEW 5 major objections 5 minor 66 references

clem:todd: A Framework for the Systematic Benchmarking of LLM-Based Task-Oriented Dialogue System Realisations

T0 review · 5 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read One self-play loop can fairly benchmark any user-simulator and dialogue-system pairing, and its results favour monolithic and LLM-modular designs over a fixed programmatic pipeline.

desk verdict Useful TOD benchmarking harness, but its architecture rankings are simulator-dependent; worth engaging as infrastructure, not as settled results. read the letter →

arxiv 2505.05445 v2 pith:PX65HWPZ submitted 2025-05-08 cs.CL

Chalamalasetti Kranti , Sherzod Hakimov , David Schlangen This is my paper · ORCID
classification cs.CL
keywords task-orienteddialogueLLMusersimulationself-playbenchmarkingclem:toddtoolcallschemabookingaccuracyMultiWOZsystemevaluation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Task-oriented dialogue systems are usually tested in isolation, so it is hard to say whether one architecture beats another because of the system or because of the test rig. This paper claims that clem:todd, a self-play benchmark built around an LLM 'user' conversing with a dialogue system under a strict JSON tool-call schema, gives a unified answer: the same dataset, metrics, and compute budget apply to any simulator–system pairing. Under that rig, the framework's experiments conclude that monolithic systems and LLM-controlled modular systems achieve higher booking accuracy than a fixed programmatic modular pipeline, at lower or comparable cost, and that the largest cause of small-model failure is format violation rather than dialogue understanding. The paper also establishes that swapping the user simulator can change a system's success rate by up to 0.58, so single-simulator scores should be read as conditional on the simulator.

What carries the argument

The load-bearing mechanism is clem:todd itself: a game-master-coordinated two-player loop in which an LLM user simulator, an LLM dialogue system, a schema-constrained tool-call interface, and automatic scoring are bound into one reproducible pipeline. The Tool Schema—a fixed set of JSON function calls such as retrievefromrestaurantdb and validatetrainbooking—is what makes the comparison clean: every system must express its behaviour as the same machine-executable actions, and any format violation ends the episode. This is the object that carries the argument, because it converts 'which architecture is better' into a question answerable under identical data, metrics, and compute for every pairing.

What would settle it

Take the same 117 booking tasks and the same three architectures, replace the LLM user simulator with transcripts from human users (or with a differently built simulator that does not defer to the system's corrections), and recompute booking accuracy. If the ranking of Monolithic, Modular-LLM, and Modular-Prog changes, or if GPT-4o's advantage over open models shrinks, the framework's architecture trade-off was driven by simulator behaviour rather than by the systems themselves.

Watch

Extended reading notes

Core claim

The discovery this paper advances is that an evaluation framework for task-oriented dialogue can be built by treating a booking task as a two-player assistance game, and that doing so makes architecture trade-offs visible and comparable. Concretely, a user simulator with a natural-language goal talks to the dialogue system; the dialogue system must reply with structured tool calls (query database, validate booking, follow up); a game master enforces the schema, aborts on any violation, executes the calls, and scores the final booking against ground truth. The paper claims this is the first setup to combine plug-and-play integration of different simulators and systems with uniform datasets, metrics, and compute constraints. Used on the MultiWOZ 2.2 booking tasks, it finds that Monolithic and Modular-LLM realisations outperform Modular-Prog on booking accuracy and cost, and that small models are defeated mainly by response-format violations rather than by dialogue failure.

Load-bearing premise

The headline rankings stand on the choice of Qwen2.5-32B as the user simulator for all the comparisons; if that simulated user is more accommodating, predictable, or goal-faithful than real customers, the winner board is an artifact of the simulator. The paper's Limitations section also flags that its modular pipelines are deliberately simple and that strict format enforcement aborts many small-model dialogues, so the scores are conditional on those design choices.

Editorial extensions

If this is right

  • Architecture rankings become commensurable: a system's score in clem:todd can be compared across papers because data, metrics, and compute are fixed by the rig rather than by each lab's choices.
  • Practitioners can pick by constraint: where compute or API cost dominates, monolithic systems are the cheaper route to high booking accuracy; where interpretability matters, Modular-LLM buys part of the benefit without Modular-Prog's cost.
  • Small-model results should be read as an instruction-following problem first: most small-model failures are aborted dialogues caused by invalid tool-call formats, so better schema adherence could raise their booking accuracy substantially.
  • Evaluators should report robustness to simulator choice rather than a single number: the observed user-spread of 0.25–0.62 means a system's reported success depends on the simulated user, and the paper recommends testing across several.
  • The rig can move to new goals: because the game logic is separate from the test instances, synthetic MultiWOZ-style and even deliberately unrealistic tasks can be injected without changing the evaluation logic.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If format violations are the dominant small-model failure, then adding constrained decoding or grammar-restricted sampling at the tool-call layer is a natural next test that clem:todd's schema makes easy; the paper does not run it, but its own data predicts a large gain for 1B–8B models.
  • The reported simulator sensitivity suggests that many published single-simulator TOD rankings may be confounded; rerunning prior systems under clem:todd with a second simulator would quantify how much of the literature's ordering is rig-dependent.
  • The unrealistic-task results imply that correcting a user's impossible request counts as a failure under strict goal matching; a separate 'goal-aware repair' metric would distinguish systems that sensibly steer the user from systems that simply lost the goal, which the paper does not compute.
  • The Turing test for simulator naturalness rested on one annotator; a multi-annotator replication with agreement statistics would be the minimal check on whether Qwen2.5-32B is genuinely more human-like, and hence on the simulator choice itself.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper proposes clem:todd, a framework for benchmarking LLM-based task-oriented dialogue systems in a self-play setup derived from clembench. A user simulator (Player A) and a dialogue system (Player B) interact through a Game Master that enforces a JSON tool schema and executes database lookups and booking validations. The framework is instantiated on a filtered MultiWOZ 2.2 test set of 117 booking tasks, using Inform and Booking Accuracy plus cost and latency metrics. The authors integrate two existing systems (AutoTOD and the Hudecek and Dusek modular pipeline) and introduce three architectures (Monolithic, Modular-Prog, Modular-LLM), evaluating them across Llama, Qwen, and GPT-4o models. They report that monolithic systems and Modular-LLM achieve higher booking accuracy at lower or comparable cost than Modular-Prog, that small models fail mainly through format violations, and that the framework can be adapted to synthetic MultiWOZ-style and unrealistic datasets.

Significance. The framework contribution is genuinely useful: it provides a unified, reproducible evaluation harness with released code, prompt templates, tool schemas, and a consistent cost model. The evaluation is anchored to an external benchmark with standard metrics, and the paper does not fit any quantity to a target result, so circularity is not a concern. If the empirical findings are robust, the comparison of monolithic versus modular control strategies and the analysis of format-violation failures would be actionable for practitioners. However, the headline architecture rankings are heavily conditioned on the choice of a single user simulator, the test set is small, and at least one appendix table is internally inconsistent. These issues make the current empirical conclusions conditional rather than established.

major comments (5)
  1. [Section 5.1.2, Tables 2 and 3] The decision to fix Qwen2.5-32B as the user simulator for all subsequent experiments is not sufficiently justified. Table 2 shows that the same dialogue system's booking accuracy varies by up to 0.58 depending on the simulator model (e.g., Qwen2.5-32B Monolithic: 0.42 with Llama-3.2-1B versus 1.00 with Llama-3.3-70B). The selection argument in Section 5.1.2 is based on naturalness/coherence scores and a Turing test, but there is no evidence that this simulator preserves the relative ordering of architectures that another simulator, or real users, would produce. Consequently, every value in Tables 3, 4, and 6 and Figure 3 is a property of the (system, simulator) pair, not of the system alone. Please either report results across the full panel of simulators, or provide rank-stability evidence showing that the architecture ordering is invariant to simulator choice, or substantially weaken the architecture-level conclusions.
  2. [Table 2 caption and Section 5.1.2] The caption of Table 2 states that the bottom row reports the 'standard deviation across user simulators', but Section 5.1.2 defines US-spread as the range (maximum minus minimum) of task success rates. The arithmetic in the table matches the range definition (e.g., 1.00 - 0.42 = 0.58 for Qwen2.5-32B Monolithic), not a standard deviation. This is an internal inconsistency in a metric that is central to the robustness analysis; please correct the caption or the definition and ensure the reported values match the stated statistic.
  3. [Appendix A.5, Table 13] The Multi-domain Modular-Prog rows in Table 13 contain identical values (0.80, 0.50, 0.45, 0.58) for four different models: Llama-3.2-1B, Llama-3.2-3B, Qwen2.5-7B, and Llama-3.1-8B. This pattern strongly suggests a copy-and-paste error. Moreover, these entries are inconsistent with Table 3: for example, Llama-3.2-1B has a Modular-Prog booking accuracy of 0.00 in Table 3, yet the single-domain and multi-domain averages in Table 13 would imply a weighted average of approximately 0.28, not 0.00; similarly, Qwen2.5-7B would imply approximately 0.52 rather than the reported 0.26. Please correct Table 13 and re-verify every appendix table against the headline results, since this undermines confidence in the domain-level analysis.
  4. [Section 5.1.3 and Figure 3] Headline comparisons such as Monolithic 0.71 versus Modular-LLM 0.68 are presented without confidence intervals or significance tests on the 117-task filtered set. With n = 117, the binomial standard error at p = 0.71 is about 0.042, so a difference of 0.03 is well within sampling noise. The larger gap to Modular-Prog (0.41) is meaningful, but the paper's claim that Modular-LLM achieves 'comparable performance' to Monolithic is not statistically supported. Please add confidence intervals or explicit uncertainty estimates, and adjust the wording so that the descriptive differences are not presented as established rankings.
  5. [Appendix A.2, Figure 7 (Prompt Template for Monolithic System)] The monolithic dialogue system prompt in Figure 7 appears to contain instructions copied from the user simulator template: it says 'Communicate naturally by expressing preferences, asking clarifying questions, and making decisions as needed' and instructs the system to 'reply with DONE' and to 'not simulate or act as the dialogue system'. If this prompt was actually used, it is inappropriate for a dialogue system and would undermine the monolith results; if it is an appendix error, the template must be corrected because the paper presents the appendix as the exact prompt used. Please clarify and fix this discrepancy.
minor comments (5)
  1. [Section 4] The filtering to 117 booking-only tasks is described briefly; please add a sentence on how this selection may affect generalizability to non-booking MultiWOZ dialogues and why the 1000-task test split was not used directly.
  2. [Table 2] The row label 'User Spread' should be made consistent with the term 'US-spread' used in Section 5.1.2 to avoid confusion.
  3. [Figure 3] The x-axis label reads 'FLOPs per token' while the surrounding text and Table 4 describe FLOP cost per dialogue; please make the units consistent.
  4. [Appendix A.6] The cost estimates depend on the assumption of $0.05 per petaFLOP and on OpenRouter prices at the time of writing; a short sensitivity discussion would be helpful since these assumptions directly drive the cost comparisons in Table 4 and Figure 3.
  5. [Section 5.1.3] The sentence 'individual modules operate without access to the full dialogue history' is presented as an explanation for modular failures; please make clear whether this is an observed property of the implementation or an interpretive claim, since the prompts in Figures 10-12 appear to include dialogue history in the input.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: system scores are computed against external MultiWOZ 2.2 ground truth, and no parameter is fitted to the headline trade-off.

full rationale

The paper's central outputs are empirical measurements, not quantities derived from their own definitions. Booking accuracy and Inform are computed by comparing dialogue-system outputs with the MultiWOZ 2.2 test-set annotations (Section 4), an external benchmark independent of clem:todd. The architecture comparison in Tables 3 and Figure 3 is therefore not equivalent to the framework's inputs by construction. The choice of Qwen2.5-32B as the default user simulator (Section 5.1.2) is an experimental design decision based on naturalness/coherence judgements and a small Turing test; while the reviewer correctly notes that all downstream rankings are conditional on this simulator and that Table 2 shows large swings in task success across simulators, that is a threat to external validity, not circularity: the reported booking rates are not defined in terms of the selection criterion, and no fitted parameter is renamed as a prediction. The self-citation to clembench (Section 3.1) is transparent and non-load-bearing: it supplies the turn-taking infrastructure, whereas the TOD-specific tool schema, booking confirmation, database retrieval, cost estimation, and synthetic-domain evaluation are new content checked against external data. The Limitations section explicitly concedes that the modular-program results may reflect pipeline design rather than model limits and that strict format enforcement may underestimate small models; these are acknowledged scope restrictions, not hidden circular assumptions. Overall, the derivation chain is self-contained with respect to the external benchmark and metrics, so the appropriate circularity score is 0.

Assumptions & free parameters 2 free parameters · 4 assumptions · 1 invented entities

The paper's central measurements rely on the MultiWOZ 2.2 ground truth, the filtered 117-task subset, LLM judges for quality scores, and the behavior of the chosen user simulator. Cost figures rest on assumed FLOP prices and OpenRouter token prices. The only invented entity is the US-spread metric, which lacks independent validation and contains a definitional inconsistency.

free parameters (2)
  • FLOP price assumption = $0.05 per petaFLOP
    Assumed from an A100 rental rate of about $2 per hour; affects all cost comparisons in Table 4 and Figure 3, but not task-success results.
  • OpenRouter token prices = As listed at the time of writing
    Used for token-cost estimates; prices drift over time and are not tied to the scientific claims about task success.
assumptions (4)
  • domain assumption MultiWOZ 2.2 ground-truth annotations are correct and the filtered 117-task subset is representative of task-oriented booking dialogues
    All Inform and Booking metrics compare system outputs to these annotations; filtering to booking tasks changes the distribution (Section 4).
  • domain assumption GPT-4o and Llama-3.3-70B zero-shot ratings of naturalness, coherence, and diversity are valid without human calibration
    Used to rank user simulators in Table 1 and to select Qwen2.5-32B for all later experiments; only 100 dialogues with one human annotator were used for a Turing test.
  • domain assumption The LLM user simulator faithfully executes the given goal and does not leak information or distort the dialogue
    The framework's task-success scoring assumes user behavior approximates a real user; Section 5.3 shows simulators adapt to unrealistic goals, which can deviate from ground truth.
  • standard math Standard LLM scaling and FLOP estimation formulas apply (2 FLOPs per parameter per token)
    Used in Appendix A.6 for cost estimates, following Kaplan et al. 2020 and Chowdhery et al. 2023.
invented entities (1)
  • US-spread robustness metric
    purpose: Quantifies the user-simulator sensitivity of a dialogue system as the range of task success across simulators
    Introduced and used in Section 5.1.2, but not validated against human judgments of robustness and internally inconsistent: defined as range, captioned as standard deviation, and the Llama Modular-Prog value 0.25 does not match either calculation.

how reviews work

0 comments
Cite this review

Pith. "Pith review of clem:todd: A Framework for the Systematic Benchmarking of LLM-Based Task-Oriented Dialogue System Realisations." pith.science (2026). https://pith.science/paper/PX65HWPZ

@misc{pith2026250505445,
  author       = {Pith},
  title        = {Pith review of: clem:todd: A Framework for the Systematic Benchmarking of LLM-Based Task-Oriented Dialogue System Realisations},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PX65HWPZ}},
  note         = {Machine review of arXiv:2505.05445}
}
read the original abstract

The emergence of instruction-tuned large language models (LLMs) has advanced the field of dialogue systems, enabling both realistic user simulations and robust multi-turn conversational agents. However, existing research often evaluates these components in isolation-either focusing on a single user simulator or a specific system design-limiting the generalisability of insights across architectures and configurations. In this work, we propose clem todd (chat-optimized LLMs for task-oriented dialogue systems development), a flexible framework for systematically evaluating dialogue systems under consistent conditions. clem todd enables detailed benchmarking across combinations of user simulators and dialogue systems, whether existing models from literature or newly developed ones. It supports plug-and-play integration and ensures uniform datasets, evaluation metrics, and computational constraints. We showcase clem todd's flexibility by re-evaluating existing task-oriented dialogue systems within this unified setup and integrating three newly proposed dialogue systems into the same evaluation pipeline. Our results provide actionable insights into how architecture, scale, and prompting strategies affect dialogue performance, offering practical guidance for building efficient and effective conversational AI systems.

Figures

Figures reproduced from arXiv: 2505.05445 by the authors.

Figure 1
Figure 1. clem:todd framework facilitates turn-based interactions between a user simulator (Player A) and a dialogue system (Player B), coordinated by a Game Master module. With the rise of LLMs, these systems (Hudecek and Dusek, 2023; Xu et al., 2024) have significantly improved in handling task-oriented, goal-driven conversations. In addition to enhancing dialogue system capabilities, LLMs now play dual roles: as end-to-end… view at source ↗
Figure 2
Figure 2. Overview of proposed dialogue system archi [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Trade-off between computational cost and [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (15 more)
Figure 4
Figure 4. Figure 4: Examples of invalid model responses due to format violations during dialogue system evaluation. [PITH_FULL_IMAGE:figures/full_fig_p015_4.png]
Figure 5
Figure 5. Figure 5: Web interface used for the Human Turing Test. Annotators were shown two dialogues side by side (a [PITH_FULL_IMAGE:figures/full_fig_p016_5.png]
Figure 6
Figure 6. Figure 6: Prompt template for the User Simulator, specifying the task description, interaction instructions, and [PITH_FULL_IMAGE:figures/full_fig_p019_6.png]
Figure 7
Figure 7. Figure 7: Prompt template for the monolithic dialogue system, detailing the task procedures, interaction instructions, [PITH_FULL_IMAGE:figures/full_fig_p020_7.png]
Figure 8
Figure 8. Figure 8: Prompt Template for the Modular-LLM Dialogue Manager, specifying task responsibilities, response [PITH_FULL_IMAGE:figures/full_fig_p021_8.png]
Figure 9
Figure 9. Figure 9: Prompt template for Dialogue Evaluation Task, describing detailed guidelines for assessing task completion, [PITH_FULL_IMAGE:figures/full_fig_p022_9.png]
Figure 10
Figure 10. Figure 10: Prompt template for the Intent Detection module, specifying task guidelines for classifying user requests [PITH_FULL_IMAGE:figures/full_fig_p023_10.png]
Figure 11
Figure 11. Figure 11: Prompt template for the Slot Extraction module, outlining task instructions for identifying and extracting [PITH_FULL_IMAGE:figures/full_fig_p024_11.png]
Figure 12
Figure 12. Figure 12: Prompt template for the Response Generation module, specifying task and response guidelines for [PITH_FULL_IMAGE:figures/full_fig_p025_12.png]
Figure 13
Figure 13. Figure 13: Schema definition detailing parameters and constraints for querying the database and booking confirma [PITH_FULL_IMAGE:figures/full_fig_p026_13.png]
Figure 14
Figure 14. Figure 14: Schema definition detailing parameters and constraints for querying the database and booking [PITH_FULL_IMAGE:figures/full_fig_p027_14.png]
Figure 15
Figure 15. Figure 15: Schema definition detailing parameters and constraints for querying the database and booking [PITH_FULL_IMAGE:figures/full_fig_p028_15.png]
Figure 16
Figure 16. Figure 16: Schema definition detailing parameters and constraints for querying the database and booking [PITH_FULL_IMAGE:figures/full_fig_p029_16.png]
Figure 17
Figure 17. Figure 17: Schema definition detailing parameters and constraints for querying the database and booking [PITH_FULL_IMAGE:figures/full_fig_p030_17.png]
Figure 18
Figure 18. Figure 18: Schema definition detailing parameters and constraints for querying the database and booking [PITH_FULL_IMAGE:figures/full_fig_p031_18.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

66 extracted references · 63 canonical work pages

  1. [1]

    System Message: Defines the role that the LLM is expected to play (e.g., user simulator, dialogue manager, slot extractor)

    work page

  2. [2]

    Task Description: Provides a brief overview of the task context and the objective the LLM should achieve

    work page

  3. [3]

    Instructions and Rules: Specifies detailed guidelines and constraints that the LLM must follow when generating responses

    work page

  4. [4]

    DARD: A Multi-Agent Approach for Task-Oriented Dialog Systems

    DARD: A multi-agent approach for task- oriented dialog systems. CoRR, abs/2411.00427. Helia Hashemi, Jason Eisner, Corby Rosset, Ben- jamin Van Durme, and Chris Kedzie. 2024. Llm- rubric: A multidimensional, calibrated approach to automated evaluation of natural language texts. In Proceedings of the 62nd Annual Meeting of the As- sociation for Computation...

    work page Pith review arXiv 2024

  5. [9]

    Single” refers to tasks within a single domain (Restaurant(R), Hotel(H), or Train(T)), while “Multi

    Test Input: The specific input query for which a response is to be generated. This prompt format ensures that models are con- textualized for their assigned tasks while allowing flexibility in content to accommodate the differing requirements of user simulation, dialogue manage- ment, and modular dialogue tasks. A.2 Existing Systems Integration As part of...

    work page 2023

  6. [20]

    Lets begin Figure 6: Prompt template for the User Simulator, specifying the task description, interaction instructions, and response format guidelines

    Use ’DONE’ only (without any prefix/suffix) to confirm task completion. Lets begin Figure 6: Prompt template for the User Simulator, specifying the task description, interaction instructions, and response format guidelines. TEMPLATE A.7.2 System Info ROLE: You are a specialized booking assistant interacting with a human user through JSON function calls us...

    work page

  7. [21]

    Extracting key details needed for the task (e.g., domain, date, time, location)

    work page

  8. [22]

    Cross-referencing user-provided information with the database to find relevant matches

    work page

Show all 66 references
  1. [23]

    If the required information is not available in the returned records, apply additional filters to narrow down the results

    If too many records are available, the database system returns only the first five. If the required information is not available in the returned records, apply additional filters to narrow down the results

  2. [24]

    Generating responses to gather missing or unclear information or to provide the booking status

  3. [25]

    For train bookings, if the database does not have trains available at the exact requested time, clarify with the user whether they are interested in seeing the closest available options that best match their query

  4. [26]

    Consolidating all extracted and clarified details for booking finalization

  5. [27]

    Keeping responses concise and focused, avoiding unnecessary elaboration or overly conversational tone

  6. [28]

    INSTRUCTIONS:

    Do not assume any details; always ask the user for clarification when necessary. INSTRUCTIONS:

  7. [29]

    Communicate naturally by expressing preferences, asking clarifying questions, and making decisions as needed

  8. [30]

    Maintain a polite and conversational tone

  9. [31]

    Do not add logic or interpretation beyond what is explicitly stated in the TASK

    Respond strictly based on the dialogue system’s response. Do not add logic or interpretation beyond what is explicitly stated in the TASK

  10. [32]

    Maintain strict adherence to the original formatting

    Ensure that names and terms remain exactly as provided in the input, without any added or altered punctuation (e.g., do not add apostrophes, hyphens, or other symbols). Maintain strict adherence to the original formatting

  11. [33]

    If the dialogue system provides alternative options close to the desired time, the you should accept a suitable nearby option that reasonably aligns with the goal

    When booking a train, exact time matches may not always be available. If the dialogue system provides alternative options close to the desired time, the you should accept a suitable nearby option that reasonably aligns with the goal

  12. [34]

    No additional text should follow/preceed

    Once the dialogue system completes the task and provides the reference number, reply with "DONE". No additional text should follow/preceed

  13. [35]

    Do not simulate or act as the dialogue system; only interact with it

  14. [36]

    OUTPUT FORMAT

    Keep responses concise and focused, avoiding unnecessary elaboration or overly conversational tone. OUTPUT FORMAT

  15. [37]

    Interaction: Respond appropriately using only the dialogue system’s response and the information under TASK

  16. [38]

    Do not add any customary comments (thank you, great etc.)

    Task Completion: Reply with "DONE". Do not add any customary comments (thank you, great etc.)

  17. [39]

    Use ’DONE’ only (without any prefix/suffix) to confirm task completion. Lets begin $USER_SIMULATOR_UTTERANCE Figure 7: Prompt template for the monolithic dialogue system, detailing the task procedures, interaction instructions, and output format for user request processing via...

  18. [40]

    Determine appropriate flow based on user input and available information b

    For each user request: a. Determine appropriate flow based on user input and available information b. Identify next required subsystem. Always use the exact subsystem names as specified in the tool schema. c. Prepare the necessary input data for that subsystem d. For database ...

  19. [41]

    Include all required fields and the response must be a valid JSON

    All responses must strictly adhere to the format. Include all required fields and the response must be a valid JSON. RESPONSE RULESS:

  20. [43]

    To interact between the sub-systems (intent detection, slot extraction, or response generation), call the ‘processnextsubsystem‘ function

  21. [44]

    To respond to the user, as a final message after coordinating with the dialogue sub-systems call the ‘followup‘ function

  22. [45]

    Similarly for booking action or database lookup, use the appropriate function from the tool schema

  23. [47]

    Never return multiple function calls in a single response

    Only one function call is allowed per turn. Never return multiple function calls in a single response. If multiple actions are needed, handle them sequentially across turns. USER REQUEST: $USER_SIMULATOR_UTTERANCE Figure 8: Prompt Template for the Modular-LLM Dialogue Manager,...

  24. [48]

    The task is completed ONLY if all the intentions are fulfilled

    Task completion You should check whether each intention in the user goal is fulfilled in the conversation. The task is completed ONLY if all the intentions are fulfilled. This would be a binary metric and you should only response with Yes or No. This would be a binary metric a...

  25. [49]

    In the dialogue, the user or the system could either be AI or human

    Naturalness This metric measures the resemblance to human. In the dialogue, the user or the system could either be AI or human. You should report a numeric rating from 1 to 5, where 5 represents most likely to be human. You are required to evaluate the naturalness of both the ...

  26. [50]

    You should report a numeric rating from 1 to 3, where 3 represents the best coherence

    Coherence This metric measures the logical consistency within a dialogue. You should report a numeric rating from 1 to 3, where 3 represents the best coherence. Here is some detailed guidelines for coherence. a. Locally, the utterances are coherent/logical based on previous tu...

  27. [51]

    For dialogue-level diversity, you only need to evaluate the user

    Dialogue-level diversity In addition to trying to achieve the initial goal, does the user introduce some reasonable deviations from the normal conversation flow? Give a score from: 3 (highest score): > 20% of the time (frequently deviate from normal flow of the conversation) 2...

  28. [52]

    booking-request: User wants to proceed with the booking

  29. [53]

    booking-success: The booking was successful and has some booking number

  30. [54]

    booking-failure: There is a failure in the booking

  31. [55]

    dbretrieval-request: User is looking for some information

  32. [56]

    dbretrieval-success: The data is fetched from the DB and the retrieval was successful

  33. [57]

    dbretrieval-failure: There is a failure in fetching the data from the DB

  34. [58]

    detection-unknown: If the input doesn’t fall into any of the above AVAILABLE DOMAINS:

  35. [59]

    Classify the request into only one of the following domains (choose the closest match): * restaurant, hotel, train

  36. [60]

    donotcare

    Not all utterances can be categorized into a domain. In such cases, use "donotcare". TASK:

  37. [61]

    Dialogue history is provided to understand the context better

    Analyze the provided input. Dialogue history is provided to understand the context better

  38. [62]

    Classify the request into only one of the above predefined intents (the closest match) and domain

  39. [63]

    Return the detected intent and domain by using those exact names

  40. [64]

    Do not add any other information or explanation or comments

  41. [66]

    Only one function call is allowed per turn. INPUT: $USER_SIMULATOR_UTTERANCE Figure 10: Prompt template for the Intent Detection module, specifying task guidelines for classifying user requests into predefined intents and domains. TEMPLATE A.7.6 System Info ROLE: You are an Sl...

  42. [67]

    Analyze the provided user request

  43. [68]

    Identify and extract relevant slots (e.g., name, area, time, date, type of cuisine, number of people, type of hotel) based on the task context

  44. [69]

    Focus on extracting the most concise and precise values for each slot, avoiding unnecessary descriptive phrases or additional words

  45. [70]

    Return the extracted slots in a structured format

  46. [71]

    Return only the formatted data—do not add explanations, comments, or additional information

  47. [72]

    Do not infer, assume, or hallucinate values based on common patterns or prior examples

    Only extract a slot if it is **explicitly mentioned** in the user input. Do not infer, assume, or hallucinate values based on common patterns or prior examples

  48. [73]

    If a relevant slot is not present in the input, **omit it from the output entirely**—do not fabricate or guess

  49. [74]

    area": "

    When handling follow-up user requests, compare the new input to the dialogue history: - If a slot was **previously extracted** but the new input **replaces or contradicts** it (e.g., rephrasing or simplifying), then **explicitly reset** that slot by setting its value to an emp...

  50. [75]

    Use the most appropriate function call based on the user’s request and available data

  51. [76]

    Never respond with plain text

    Every response MUST be a valid tool call (tool_call). Never respond with plain text

  52. [77]

    Only one function call is allowed per turn. USER REQUEST: $USER_SIMULATOR_UTTERANCE Figure 11: Prompt template for the Slot Extraction module, outlining task instructions for identifying and extracting structured key entities from user requests. TEMPLATE A.7.7 System Info ROLE...

  53. [78]

    If additional information is required to proceed, respond conversationally using direct and focused phrasing

    Generate a meaningful response: a. If additional information is required to proceed, respond conversationally using direct and focused phrasing. b. If recommendations are provided in the DB: * Ask the user to choose from the list of options. * Clearly present all options to th...

  54. [79]

    type": "function

    Guidelines for Response: a. Responses must be concise and to the point. b. Avoid unnecessary elaboration or an overly conversational tone. c. Do not generate or fabricate any information that is not explicitly present in the DB or provided input. d. If too many records are ava...

  55. [2020]

    In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2877–2888, Online

    How to make neural natural language genera- tion as reliable as templates in task-oriented dialogue. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2877–2888, Online. Association for Computa- tional Linguistics. Mihail ...

    2020

  56. [2023]

    CoRR, abs/2309.13233

    User simulation with large language mod- els for evaluating task-oriented dialogue. CoRR, abs/2309.13233. Wenjie Dong, Sirong Chen, and Yan Yang. 2025. Pro- tod: Proactive task-oriented dialogue system based on large language model. In Proceedings of the 31st International Con...

    2025 arXiv

  57. [2024]

    Preprint, arXiv:2407.21783

    The llama 3 herd of models. Preprint, arXiv:2407.21783. Leon Guertler, Bobby Cheng, Simon Yu, Bo Liu, Leshem Choshen, and Cheston Tan. 2025. Textarena. arXiv preprint arXiv:2504.11442. Aman Gupta, Anirudh Ravichandran, Ziji Zhang, Swair Shah, Anurag Beniwal, and Narayanan Sadagopan

    2025 arXiv

  58. [2763]

    Yi-Ting Yeh, Maxine Eskénazi, and Shikib Mehri

    Association for Computational Linguistics. Yi-Ting Yeh, Maxine Eskénazi, and Shikib Mehri. 2021. A comprehensive assessment of dialog evaluation metrics. CoRR, abs/2106.03706. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuoha...

    2021 arXiv

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.