pith. sign in

arxiv: 1907.05346 · v1 · pith:2DKRYVYUnew · submitted 2019-07-10 · 💻 cs.CL · cs.IR· cs.LG

A Modular Task-oriented Dialogue System Using a Neural Mixture-of-Experts

Jiahuan Pei , Pengjie Ren , Maarten de Rijke This is my paper

Pith reviewed 2026-05-24 23:50 UTC · model grok-4.3

classification 💻 cs.CL cs.IRcs.LG
keywords task-oriented dialoguemixture of expertsmodular dialogue systemneural response generationend-to-end trainingdialogue benchmarksinform ratesuccess rate
0
0 comments X

The pith

A chair bot coordinating specialized expert bots via token-level mixture improves task-oriented dialogue inform rate by 8.1% over single models.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper proposes dividing response generation in task-oriented dialogue systems across multiple expert bots, each specialized for situations such as a particular domain or action type, with a chair bot that selects the appropriate expert for the current context. This modular setup is realized through the TokenMoE model, in which all experts produce token predictions at each step and the chair combines them to form the final output. The entire system trains end-to-end on a benchmark dataset, yielding an 8.1% gain in inform rate and 0.8% gain in success rate relative to conventional single-module models.

Core claim

The authors introduce the MTDS framework consisting of a chair bot and several expert bots, implemented by the TokenMoE model where expert bots predict multiple tokens at each timestamp and the chair bot selects the final token after considering all expert outputs; both chair and experts are trained jointly in an end-to-end manner, producing an 8.1% improvement in inform rate and 0.8% improvement in success rate on a benchmark dataset compared with a single-module baseline.

What carries the argument

Token-level Mixture-of-Expert (TokenMoE) model, in which expert bots each predict tokens and the chair bot determines the output token from the full set of expert predictions.

If this is right

  • Specialization to domains or action types enables more effective responses to varied and complex dialogue contexts.
  • Joint end-to-end training lets the chair bot learn expert selection directly from the final performance objective.
  • Token-level combination allows mixing of expert outputs within a single response rather than committing to one expert for an entire turn.
  • The framework supports incremental addition of new expert bots for new situations while keeping the rest of the system intact.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Token-level mixing may permit the system to blend contributions from multiple experts inside one response instead of selecting a single expert for the whole utterance.
  • The approach could extend to multi-domain or rapidly shifting contexts where different experts become relevant at different moments.
  • If selection remains accurate at scale, the modular design might reduce reliance on manually engineered dialogue policies.

Load-bearing premise

The chair bot can reliably select the correct expert bot for each dialogue context without its selection errors canceling the performance gains from specialization.

What would settle it

Measure the chair bot's expert-selection accuracy on held-out dialogues and test whether higher selection accuracy produces the full reported gains in inform and success rates while lower accuracy produces smaller or zero gains.

Figures

Figures reproduced from arXiv: 1907.05346 by Jiahuan Pei, Maarten de Rijke, Pengjie Ren.

Figure 1
Figure 1. Figure 1: Modular Task-oriented Dialogue System (MTDS) framework. bot is specialized for a particular situation, e.g., one domain, one type of action of a system, etc. The chair bot coordinates multiple expert bots and adaptively selects an expert bot to generate the final response. Compared with existing end-to-end single-module TDSs, the advantages of MTDSs are two-fold. First, the specializa￾tion of different exp… view at source ↗
Figure 2
Figure 2. Figure 2: Overview of TokenMoE. Figure (a) illustrates how does the model generate the token [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
read the original abstract

End-to-end Task-oriented Dialogue Systems (TDSs) have attracted a lot of attention for their superiority (e.g., in terms of global optimization) over pipeline modularized TDSs. Previous studies on end-to-end TDSs use a single-module model to generate responses for complex dialogue contexts. However, no model consistently outperforms the others in all cases. We propose a neural Modular Task-oriented Dialogue System(MTDS) framework, in which a few expert bots are combined to generate the response for a given dialogue context. MTDS consists of a chair bot and several expert bots. Each expert bot is specialized for a particular situation, e.g., one domain, one type of action of a system, etc. The chair bot coordinates multiple expert bots and adaptively selects an expert bot to generate the appropriate response. We further propose a Token-level Mixture-of-Expert (TokenMoE) model to implement MTDS, where the expert bots predict multiple tokens at each timestamp and the chair bot determines the final generated token by fully taking into consideration the outputs of all expert bots. Both the chair bot and the expert bots are jointly trained in an end-to-end fashion. To verify the effectiveness of TokenMoE, we carry out extensive experiments on a benchmark dataset. Compared with the baseline using a single-module model, our TokenMoE improves the performance by 8.1% of inform rate and 0.8% of success rate.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 1 minor

Summary. The paper proposes a Modular Task-oriented Dialogue System (MTDS) framework consisting of a chair bot that adaptively selects among multiple specialized expert bots, implemented via a Token-level Mixture-of-Experts (TokenMoE) model in which experts predict tokens at each step and the chair combines their outputs. All components are jointly trained end-to-end. On a benchmark dataset, TokenMoE is reported to improve inform rate by 8.1% and success rate by 0.8% relative to a single-module baseline.

Significance. If the performance deltas are robust and can be attributed to the modular specialization rather than capacity or training effects, the approach could help address the observation that no single end-to-end model dominates all dialogue contexts. The joint training of chair and experts is a constructive design choice that avoids the need for separate pre-training stages.

major comments (2)
  1. [Abstract] Abstract: the headline claim of an 8.1% inform-rate and 0.8% success-rate lift is presented without any information on dataset size, baseline architecture details, number of runs, variance, or statistical testing, so the reliability of the central empirical result cannot be assessed from the supplied evidence.
  2. [Experiments] Experiments (implied by the abstract's comparison): no ablation, oracle-chair experiment, or chair-selection accuracy metric is described that would isolate whether the adaptive selection mechanism contributes net value or whether selection errors offset the gains from expert specialization, leaving the load-bearing assumption that modularity is responsible for the deltas untested.
minor comments (1)
  1. [Abstract] The abstract would be clearer if it named the specific benchmark dataset and the single-module baseline architecture.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive feedback. We address each major comment below and indicate the revisions planned for the manuscript.

read point-by-point responses

  1. Referee: [Abstract] Abstract: the headline claim of an 8.1% inform-rate and 0.8% success-rate lift is presented without any information on dataset size, baseline architecture details, number of runs, variance, or statistical testing, so the reliability of the central empirical result cannot be assessed from the supplied evidence.

    Authors: We agree that the abstract would benefit from additional context. The manuscript body describes the benchmark dataset and single-module baselines, but the original submission does not report number of runs, variance, or statistical testing. We will revise the abstract to include dataset size, baseline architecture details, and a summary of the experimental protocol. revision: yes

  2. Referee: [Experiments] Experiments (implied by the abstract's comparison): no ablation, oracle-chair experiment, or chair-selection accuracy metric is described that would isolate whether the adaptive selection mechanism contributes net value or whether selection errors offset the gains from expert specialization, leaving the load-bearing assumption that modularity is responsible for the deltas untested.

    Authors: We acknowledge that the original manuscript does not include ablations, oracle-chair experiments, or chair-selection accuracy metrics. The reported gains are relative to single-module baselines, but these additional experiments would more directly test the contribution of adaptive selection. We will add an ablation study and an oracle-chair experiment to the revised manuscript. revision: yes

Circularity Check

0 steps flagged

No circularity; empirical gains reported against external baseline on benchmark data

full rationale

The paper introduces an MTDS framework and TokenMoE implementation, describes joint end-to-end training of chair and expert components, and states empirical improvements (8.1% inform rate, 0.8% success rate) versus a single-module baseline on a benchmark dataset. No equations, fitted parameters, or self-citations appear in the provided text that would reduce any claimed result to an input by construction. The performance delta is presented as an experimental outcome rather than a definitional or fitted tautology, satisfying the criteria for a self-contained result against external benchmarks.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

Abstract-only review supplies no explicit free parameters, axioms, or invented entities; the central claim rests on the unstated premise that expert specialization plus chair selection yields net gains.

pith-pipeline@v0.9.0 · 5796 in / 1071 out tokens · 17248 ms · 2026-05-24T23:50:07.447794+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

35 extracted references · 35 canonical work pages

  1. [1]

    Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In International Conference on Learning Representations (ICLR ’15). –

    work page 2015

  2. [2]

    Ankur Bapna, Gokhan Tur, Dilek Hakkani-Tur, and Larry Heck. 2017. Sequential dialogue context modeling for spoken language understanding. In Proceedings of the 18th Annual SIGdial Meeting on Discourse and Dialogue (SIGDIAL ’17) . 103–114

    work page 2017

  3. [3]

    Antoine Bordes and Jason Weston. 2017. Learning end-to-end goal-oriented dialog. In International Conference on Learning Representations (ICLR ’17). –

    work page 2017

  4. [4]

    Pawel Budzianowski, Iñigo Casanueva, Bo-Hsiang Tseng, and Milica Gasic. 2018. Towards end-to-end multi-domain dialogue modelling . Technical Report. Cam- bridge University

    work page 2018

  5. [5]

    Paweł Budzianowski, Tsung-Hsien Wen, Bo-Hsiang Tseng, Iñigo Casanueva, Stefan Ultes, Osman Ramadan, and Milica Gasic. 2018. MultiWOZ-A large- scale multi-domain wizard-of-oz dataset for task-oriented dialogue modelling. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (EMNLP ’18). 5016–5026

    work page 2018

  6. [6]

    Hongshen Chen, Xiaorui Liu, Dawei Yin, and Jiliang Tang. 2017. A survey on dialogue systems: Recent advances and new frontiers. ACM SIGKDD Explorations Newsletter 19, 2 (2017), 25–35

    work page 2017

  7. [7]

    Po-Chun Chen, Ta-Chung Chi, Shang-Yu Su, and Yun-Nung Chen. 2017. Dynamic time-aware attention to speaker roles and contexts for spoken language under- standing. In Proceedings of 2017 IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU ’17). 554–560

    work page 2017

  8. [8]

    Paul Crook, Alex Marin, Vipul Agarwal, Khushboo Aggarwal, Tasos Anastasakos, Ravi Bikkula, Daniel Boies, Asli Celikyilmaz, Senthilkumar Chandramohan, Zhaleh Feizollahi, et al . 2016. Task completion platform: A self-serve multi- domain goal oriented dialogue platform. In Proceedings of the 2016 Conference of the North American Chapter of the Association f...

    work page 2016

  9. [9]

    Thomas G Dietterich. 2000. Ensemble methods in machine learning. In Proceed- ings of the First International Workshop on Multiple Classifier Systems (MCS ’00) . 1–15

    work page 2000

  10. [10]

    Ondrej Dušek and Filip Jurcıcek. 2016. A context-aware natural language genera- tor for dialogue systems. In Proceedings of the 17th Annual Meeting of the Special Interest Group on Discourse and Dialogue (SIGDIAL ’16) . 185–190

    work page 2016

  11. [11]

    Sergey Edunov, Myle Ott, Michael Auli, and David Grangier. 2018. Understanding back-translation at scale. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (EMNLP ’18) . 489–500

    work page 2018

  12. [12]

    Mihail Eric, Lakshmi Krishnan, Francois Charette, and Christopher D Manning

    work page

  13. [13]

    In Proceedings of the 18th Annual Meeting of the Special Interest Group on Discourse and Dialogue (SIGDIAL ’17)

    Key-value retrieval networks for task-oriented dialogue. In Proceedings of the 18th Annual Meeting of the Special Interest Group on Discourse and Dialogue (SIGDIAL ’17). 37–49

    work page

  14. [14]

    Jiang Guo, Darsh J Shah, and Regina Barzilay. 2018. Multi-source domain adapta- tion with mixture of experts. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (EMNLP ’18) . 4694–4703

    work page 2018

  15. [15]

    Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory.Neural Computation 9, 8 (1997), 1735–1780

    work page 1997

  16. [16]

    Diederik Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimiza- tion. In International Conference on Learning Representations (ICLR ’15). –

    work page 2015

  17. [17]

    Wenqiang Lei, Xisen Jin, Min-Yen Kan, Zhaochun Ren, Xiangnan He, and Dawei Yin. 2018. Sequicity: Simplifying task-oriented dialogue systems with single sequence-to-sequence architectures. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (ACL ’18). 1437–1447

    work page 2018

  18. [18]

    Jiwei Li, Michel Galley, Chris Brockett, Georgios P Spithourakis, Jianfeng Gao, and Bill Dolan. 2016. A persona-based neural conversation model. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (ACL ’16). 994–1003

    work page 2016

  19. [19]

    Thang Luong, Hieu Pham, and Christopher D. Manning. 2015. Effective ap- proaches to attention-based neural machine translation. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing (EMNLP ’15). 1412–1421

    work page 2015

  20. [20]

    Saeed Masoudnia and Reza Ebrahimpour. 2014. Mixture of experts: a literature survey. Artificial Intelligence Review 42, 2 (2014), 275–293

    work page 2014

  21. [21]

    Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. 2013. On the difficulty of training recurrent neural networks. In Proceedings of the 30th International Conference on Machine Learning (ICML ’13). 1310–1318

    work page 2013

  22. [22]

    Abhinav Rastogi, Raghav Gupta, and Dilek Hakkani-Tur. 2018. Multi-task learning for joint language understanding and dialogue state tracking. InProceedings of the 19th Annual SIGdial Meeting on Discourse and Dialogue (SIGDIAL ’19) . 376–384

    work page 2018

  23. [23]

    Patrick Schwab, Djordje Miladinovic, and Walter Karlen. 2019. Granger-causal attentive mixtures of experts: Learning important features with neural networks. In AAAI Conference on Artificial Intelligence (AAAI ’19). –

    work page 2019

  24. [24]

    Iulian Vlad Serban, Alessandro Sordoni, Yoshua Bengio, Aaron C Courville, and Joelle Pineau. 2016. Building end-to-end dialogue systems using generative hierarchical neural network models. In Thirtieth AAAI Conference on Artificial Intelligence (AAAI ’16). 3776–3784

    work page 2016

  25. [25]

    Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. 2017. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. In International Conference on Learning Representations (ICLR ’17). –

    work page 2017

  26. [26]

    Alessandro Sordoni, Michel Galley, Michael Auli, Chris Brockett, Yangfeng Ji, Margaret Mitchell, Jian-Yun Nie, Jianfeng Gao, and Bill Dolan. 2015. A neural network approach to context-sensitive generation of conversational responses. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human...

    work page 2015

  27. [27]

    Oriol Vinyals and Quoc Le. 2015. A neural conversational model. In ICML Deep Learning Workshop. –

    work page 2015

  28. [28]

    Tsung-Hsien Wen, Milica Gasic, Nikola Mrksic, Pei-Hao Su, David Vandyke, and Steve Young. 2015. Semantically conditioned lstm-based natural language generation for spoken dialogue systems. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing (EMNLP ’15) . 1711–1721

    work page 2015

  29. [30]

    Tsung-Hsien Wen, David Vandyke, Nikola Mrkšić, Milica Gasic, Lina M Rojas Barahona, Pei-Hao Su, Stefan Ultes, and Steve Young. 2017. A network-based end-to-end trainable task-oriented dialogue system. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics (EACL ’17). 438–449

    work page 2017

  30. [31]

    Jason D Williams, Kavosh Asadi, and Geoffrey Zweig. 2017. Hybrid code net- works: practical and efficient end-to-end dialog control with supervised and reinforcement learning. In Proceedings of the 55th Annual Meeting of the Associa- tion for Computational Linguistics (ACL ’17). 665–677

    work page 2017

  31. [32]

    Zhao Yan, Nan Duan, Peng Chen, Ming Zhou, Jianshe Zhou, and Zhoujun Li

    work page

  32. [33]

    InThirty-First AAAI Conference on Artificial Intelligence (AAAI ’2017)

    Building task-oriented dialogue systems for online shopping. InThirty-First AAAI Conference on Artificial Intelligence (AAAI ’2017). 4618–4626

    work page 2017

  33. [34]

    Sanghyun Yi, Rahul Goel, Chandra Khatri, Tagyoung Chung, Behnam Hedayatnia, Anu Venkatesh, Raefer Gabriel, and Dilek Hakkani-Tur. 2019. Towards coherent and engaging spoken dialog response generation using automatic conversation evaluators. arXiv preprint arXiv:1904.13015 (2019)

    work page arXiv 2019

  34. [35]

    Steve Young, Milica Gašić, Blaise Thomson, and Jason D Williams. 2013. POMDP- based statistical spoken dialog systems: A review. Proc. IEEE 101, 5 (2013), 1160–1179

    work page 2013

  35. [36]

    Victor Zhong, Caiming Xiong, and Richard Socher. 2018. Global-locally self- attentive encoder for dialogue state tracking. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (ACL ’18) . 1458–1467

    work page 2018