REVIEW 2 major objections 5 minor 29 references
Scalable and Accurate Dialogue State Tracking via Hierarchical Sequence Generation
T0 review · 2 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash
Pith's one-line read A dialogue state tracker that generates belief states in constant time, independent of ontology size, reaches state-of-the-art multi-domain accuracy.
desk verdict A solid DST paper with a genuinely new hierarchical generation idea, but the central O(1) complexity claim is true only under a narrow definition and the empirical confirmation is weaker than presented. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The Conditional Memory Relation Decoder (CMRD) is the central mechanism. It is a recurrent decoder that, at each step, takes a condition vector (the parent representation from the hierarchy), attends to encoded belief-state, system, and user memories in sequence, concatenates the attention outputs into a working memory, and passes that through a multi-layer perceptron for relation reasoning. The same CMRD is reused for domain, slot, and value generation, which is what makes the hierarchy depth flexible and the parameter count independent of ontology size. The model also uses pretrained transformer embeddings and byte-pair encoding so that unseen words can be represented and generated.
What would settle it
Measure the model's wall-clock inference time on dialogues with the same number of active slots but with increasingly large predefined ontologies (e.g., adding thousands of unused values); if runtime grows linearly with the ontology size or with the generated sequence length, the O(1) complexity claim, taken as compute, is false.
Extended reading notes
Core claim
The central claim is that dialogue state tracking can be solved as a sequence-to-sequence problem without enumerating ontology candidates. COMER encodes the user utterance, previous system actions, and previous belief state, then uses three hierarchically stacked decoders that share all parameters: the first generates the sequence of domains, the second generates the slots for each generated domain, and the third generates the value for each generated slot. Because the model generates rather than scores domain-slot-value triples, its inference time complexity is O(1) with respect to the number of predefined domains, slots, and values. Empirically, the paper shows that the wall-clock multiplier from WoZ2.0 to MultiWOZ is close to the theoretical multiplier predicted for an O(1) model, and that the joint goal accuracy on MultiWOZ, 48.79%, is state of the art at publication time.
Load-bearing premise
The constant-time claim counts the number of inference passes per turn, not the actual compute, which still includes a softmax over the full predefined domain and slot vocabulary and a generation length that can grow with the number of active slots.
Editorial extensions
If this is right
- Dialogue systems can be extended to new domains without rebuilding a candidate list or re-scoring thousands of slot-value pairs at each turn.
- The joint goal accuracy on MultiWOZ indicates that generation-based trackers can match or exceed classification-based trackers on multi-domain benchmarks.
- The parameter-sharing across hierarchical levels suggests the model can in principle handle deeper or more complex belief structures, such as values with multiple alternatives, without architectural changes.
- The reported 2.54x actual runtime multiplier when moving from WoZ2.0 to the five-times-larger MultiWOZ is consistent with the paper's claim of constant inference passes per turn.
- Generation-based tracking can produce structured belief states that are not constrained by a predefined ontology, which may help downstream modules in open-vocabulary dialogue settings.
Reading between the lines
- If the O(1) claim is interpreted as counting decoder passes, the practical compute still scales with the vocabulary size of the final softmax and with the number of generated tokens; a stricter test would measure end-to-end latency against ontology size while holding dialogue content fixed.
- The same hierarchical generation idea could transfer to other structured prediction problems, such as nested information extraction or hierarchical multi-label classification, where the output has a parent-child relation.
- The paper's slot-prediction weakness (58.43% slot accuracy given the correct domain) suggests that richer dialogue-history representations, rather than larger candidate sets, are the next likely source of accuracy gains.
- A testable extension is to replace the frequency-based ordering of domains and slots with an adaptive or learned ordering, which could affect both accuracy and the length of generated sequences.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes COMER, a hierarchical sequence-generation formulation of dialogue state tracking in which a shared Conditional Memory Relation decoder first generates the sequence of domains, then slots, then values for each turn. The authors claim that this removes the dependence of inference time on the pre-defined ontology and that COMER therefore has O(1) inference-time complexity with respect to the number of domains, slots, and values. Experiments on WoZ2.0 and MultiWOZ report joint-goal accuracies of 88.6% and 48.79%, respectively, the latter being a marginal improvement over TRADE, together with ablations of the attention hierarchy, MLP, parameter sharing, ordering, nested representation, and gradient blocking. The code is released publicly.
Significance. If the constant-complexity claim were accurate, COMER would be the first DST with ontology-independent inference cost while matching state-of-the-art accuracy, which would be a meaningful contribution to scalable task-oriented dialogue. The paper also makes useful empirical contributions: it provides a released implementation, a careful multi-level ablation on MultiWOZ (joint domain, domain-slot, and joint goal accuracy), and a sequence-generation view of belief states that is natural for nested domain-slot-value structures. The SOTA-level joint goal accuracy of 48.79% on MultiWOZ is itself plausible and valuable, and the qualitative attention analysis is informative. However, the central scalability claim is substantially overstated under a conventional definition of computational complexity, and the reported empirical confirmation in Section 4.3 does not isolate the dependence on ontology size.
major comments (2)
- [Section 3.2 and Abstract/Conclusion] The O(1) inference-time claim is only valid under the narrow definition of 'how many times inference must be performed' in Table 1, i.e., one decoder invocation per turn. The Abstract and Conclusion instead claim constant computational complexity, which is not supported. Per generated token, the decoder computes ps = softmax(E^T h_o) over the full static embedding matrix E, and Section 3.1 defines E as the concatenation of the BERT vocabulary Ev with the extra slot embedding Es, which contains one embedding for every domain/slot seen in training. Thus the per-step softmax cost grows with the number of pre-defined labels. In addition, the number of generated tokens per turn is the length of the belief-state sequence, which can grow with the number of active slots and, in the worst case, with n. The paper should either explicitly restrict the claim to 'one decoder call per turn' and qualify all downstream statements, or provide a per-step complexity analysis that includes the output-vocabulary dimension and the autoregressive sequence length.
- [Section 4.3] The empirical confirmation of O(1) complexity is not convincing because the theoretical multiplier K is computed under the assumption h(n)=h(m)=1, i.e., the O(1) class is already assumed when defining K. The observed 2.54x multiplier is obtained by comparing two datasets that differ simultaneously in the number of slots (3 vs 35), number of values (99 vs 4510), average turns per dialogue, average tokens per turn, and validation-set size, so the comparison does not isolate the dependence on n or m. The match between 2.54 and 2.15 is a consistency check of the overall runtime model, not a test of O(1) with respect to ontology size. To support the scalability claim, the authors should report a controlled experiment (for example, varying the number of pre-defined slots on a fixed test set while measuring wall-clock time) or at least report per-token decoding time and the distribution of generated belief-state lengths on MultiWOZ.
minor comments (5)
- [Table 1 caption] The caption sentence 'The ITC is calculated based on how many times inference must be performed to complete a prediction of the belief state in a dialogue turn' is not a complete sentence; it should be integrated into a full sentence describing the definition of ITC.
- [Section 3.2] The statement that 'the model parameters are independent of the vocabulary size' is correct but should not be conflated with computational independence: the softmax operation in ps = softmax(E^T h_o) still iterates over the full output vocabulary, including the Es component.
- [Table 4] The label 'Hierachical-Attn' contains a typo and should read 'Hierarchical-Attn'.
- [Section 4.3] The phrase 'roughly of the same magnitude as the theoretical value of 2.15' should be stated with a numerical margin or confidence interval; the reported 2.54 is 18% larger than 2.15, and without additional analysis it is unclear whether this difference is expected under the model's assumptions.
- [Section 4.4] The sentence 'The effectiveness of our hierarchical attention design is proved by an accuracy drop of 1.95%' should be softened to 'is supported by', since a single ablation is suggestive rather than a proof.
Circularity Check
No significant circularity: the O(1) claim is explicitly grounded in the paper's ITC definition (one inference pass per turn), and the accuracy results are external-benchmark evaluations; Section 4.3 is a confounded confirmation but not a derivation that reduces to its own input.
full rationale
COMER's central O(1) claim is not obtained by fitting a parameter and then predicting a closely related quantity. Table 1 defines ITC as the number of times inference must be performed to complete a belief-state prediction in a turn. Section 3 explicitly states that COMER generates the belief state as one hierarchical sequence per turn rather than iterating over predefined slots, so under that stated metric the O(1) ITC follows from the model design. The joint-goal accuracy results in Table 3 are measured on held-out WoZ2.0 and MultiWOZ test sets against published baselines, so they are independent external benchmarks rather than self-referential outputs. The only potentially circular-looking passage is Section 4.3, where the theoretical ITM is computed by setting h(n)=h(m)=1 for an O(1) model (giving 2.15x), and the measured 2.54x runtime ratio is then described as empirical confirmation. This is a weak and confounded test because the two datasets differ in turn length, token counts, vocabulary, and validation-set size, and the decoder's per-step softmax in Section 3.2 still ranges over static embeddings that include domain/slot entries. However, the measured runtime is empirical and is not algebraically forced by the theoretical multiplier; the argument is a limitation of the evidence, not a circular reduction. The self-citation of Ren et al. (2018) is used only as a baseline and to justify dropping the rare name slot, so it is not load-bearing. No step meets the threshold of a prediction being equivalent to its inputs by construction. Score 0.
Assumptions & free parameters
free parameters (6)
- dropout rate p =
0.5
- model size dm =
512
- embedding size de =
1024
- learning rate =
0.0005
- MLP layers =
4 (selected by grid search)
- batch size =
32 (WoZ), 16 (MultiWOZ)
assumptions (4)
- domain assumption BERT provides fixed contextual embeddings and static word embeddings.
- domain assumption Domains and slots from the training set are a closed vocabulary for generation.
- domain assumption Belief states can be linearized into a frequency-ordered sequence without loss of information.
- domain assumption The single value assumption is invalid for some dialogues, but the model is trained and evaluated on standard single-valued labels.
Cite this review
Pith. "Pith review of Scalable and Accurate Dialogue State Tracking via Hierarchical Sequence Generation." pith.science (2026). https://pith.science/paper/A5MX7CHZ
@misc{pith2026190900754,
author = {Pith},
title = {Pith review of: Scalable and Accurate Dialogue State Tracking via Hierarchical Sequence Generation},
year = {2026},
howpublished = {\url{https://pith.science/paper/A5MX7CHZ}},
note = {Machine review of arXiv:1909.00754}
}
read the original abstract
Existing approaches to dialogue state tracking rely on pre-defined ontologies consisting of a set of all possible slot types and values. Though such approaches exhibit promising performance on single-domain benchmarks, they suffer from computational complexity that increases proportionally to the number of pre-defined slots that need tracking. This issue becomes more severe when it comes to multi-domain dialogues which include larger numbers of slots. In this paper, we investigate how to approach DST using a generation framework without the pre-defined ontology list. Given each turn of user utterance and system response, we directly generate a sequence of belief states by applying a hierarchical encoder-decoder structure. In this way, the computational complexity of our model will be a constant regardless of the number of pre-defined slots. Experiments on both the multi-domain and the single domain dialogue state tracking dataset show that our model not only scales easily with the increasing number of pre-defined domains and slots but also reaches the state-of-the-art performance.
Figures
Figures from the paper (2 more)
Reference graph
Works this paper leans on
-
[1]
URL: " 'urlintro :=
ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRINGS urlintro eprinturl eprintpr...
-
[2]
write newline
" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...
-
[3]
Pawel Budzianowski, Tsung-Hsien Wen, Bo-Hsiang Tseng, I \ n igo Casanueva, Stefan Ultes, Osman Ramadan, and Milica Gasic. 2018. Multiwoz - a large-scale multi-domain wizard-of-oz dataset for task-oriented dialogue modelling. In EMNLP
work page 2018
-
[4]
Kyunghyun Cho, Bart van Merrienboer, aglar Gülehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning phrase representations using rnn encoder-decoder for statistical machine translation. In EMNLP
work page 2014
-
[5]
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. In NAACL-HLT
2018
-
[6]
Shuyang Gao, Abhishek Sethi, Sanchit Agarwal, Tagyoung Chung, and Dilek Zeynep Hakkani. 2019. Dialog state tracking: A neural reading comprehension approach. ArXiv, abs/1908.01946
work page Pith review arXiv 2019
-
[7]
Rahul Goel, Shachi Paul, and Dilek Zeynep Hakkani. 2019. Hyst: A hybrid approach for flexible and accurate dialogue state tracking. ArXiv, abs/1907.00883
arXiv 2019
-
[8]
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2015. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. 2015 IEEE International Conference on Computer Vision (ICCV), pages 1026--1034
work page 2015
Show all 29 references
-
[9]
Williams
Matthew Henderson, Blaise Thomson, and Jason D. Williams. 2014. The second dialog state tracking challenge. In SIGDIAL Conference
2014
-
[10]
Sepp Hochreiter and J \"u rgen Schmidhuber. 1997. Long short-term memory. Neural Computation, 9:1735--1780
1997
-
[11]
Kingma and Jimmy Ba
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. CoRR
2015
-
[12]
Fei Liu and Julien Perez. 2017. Dialog state tracking, a machine reading approach using memory network. In EACL
2017
-
[13]
Nikola Mrksic, Diarmuid \'O S \'e aghdha, Tsung-Hsien Wen, Blaise Thomson, and Steve J. Young. 2017. Neural belief tracker: Data-driven dialogue state tracking. In ACL
2017
-
[14]
Elnaz Nouri and Ehsan Hosseini-Asl. 2018. Toward scalable neural dialogue state tracking model. arXiv preprint arXiv:1812.00899
2018 arXiv
-
[15]
Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke S
Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke S. Zettlemoyer. 2018. Deep contextualized word representations. In NAACL-HLT
2018
-
[16]
Osman Ramadan, Pawel Budzianowski, and Milica Gasic. 2018. Large-scale multi-domain belief tracking with knowledge sharing. In ACL
2018
-
[17]
Abhinav Rastogi, Dilek Hakkani-Tur, and Larry Heck. 2017. Scalable multi-domain dialogue state tracking. arXiv preprint arXiv:1712.10224
2017 arXiv
-
[18]
Reddi, Satyen Kale, and Sanjiv Kumar
Sashank J. Reddi, Satyen Kale, and Sanjiv Kumar. 2018. On the convergence of adam and beyond. In ICLR
2018
-
[19]
Liliang Ren, Kaige Xie, Lu Chen, and Kai Yu. 2018. Towards universal dialogue state tracking. In EMNLP
2018
-
[20]
Adam Santoro, David Raposo, David G. T. Barrett, Mateusz Malinowski, Razvan Pascanu, Peter W. Battaglia, and Timothy P. Lillicrap. 2017. A simple neural network module for relational reasoning. In NIPS
2017
-
[21]
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. CoRR, abs/1508.07909
2016 arXiv
-
[22]
Sainbayar Sukhbaatar, Arthur Szlam, Jason Weston, and Rob Fergus. 2015. End-to-end memory networks. In NIPS
2015
-
[23]
Tsung-Hsien Wen, Milica Gasic, Nikola Mrksic, Lina Maria Rojas-Barahona, Pei hao Su, Stefan Ultes, David Vandyke, and Steve J. Young. 2017. A network-based end-to-end trainable task-oriented dialogue system. In EACL
2017
-
[24]
Chien-Sheng Wu, Andrea Madotto, Ehsan Hosseini-Asl, Caiming Xiong, Richard Socher, and Pascale Fung. 2019. Transferable multi-domain state generator for task-oriented dialogue systems. In ACL
2019
-
[25]
Puyang Xu and Qi Hu. 2018. An end-to-end approach for handling unknown slot values in dialogue state tracking. In ACL
2018
-
[26]
Pengcheng Yang, Xu Sun, Wei Li, Shuming Ma, Wei Wu, and Houfeng Wang. 2018. Sgm: Sequence generation model for multi-label classification. In COLING
2018
-
[27]
Steve Young, Milica Ga s i \'c , Blaise Thomson, and Jason D Williams. 2013. Pomdp-based statistical spoken dialog systems: A review. Proceedings of the IEEE, 101(5):1160--1179
2013
-
[28]
Yulun Zhang, Yapeng Tian, Yu Kong, Bineng Zhong, and Yun Fu. 2018. Residual dense network for image super-resolution. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2472--2481
2018
-
[29]
Victor Zhong, Caiming Xiong, and Richard Socher. 2018. Global-locally self-attentive dialogue state tracker. CoRR, abs/1805.09655
2018 arXiv
Reviewed August 14, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.