{"id":"ffb01b3d-39c9-48b6-88c8-e6ad71888171","arxiv_id":"1909.00754","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"COMER generates belief states hierarchically with a shared sequence decoder, achieving 48.79% joint goal accuracy on MultiWOZ and near-state-of-the-art on WoZ2.0, while claiming O(1) inference time relative to the predefined ontology.","lead":"COMER is a dialogue state tracker that generates belief states as hierarchical sequences of domains, slots, and values, rather than scoring each predefined slot. The paper reports joint goal accuracy of 48.79% on MultiWOZ and claims constant inference time with respect to the ontology size.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The O(1) scalability claim is an artifact of counting one high-level decoder call per turn; per-step softmax cost over an ontology-dependent output matrix and autoregressive output length reintroduce dependence on the ontology and on active-slot count.","rationale":"The reader's weakest assumption identifies exactly the same load-bearing weakness: the O(1) claim is stated in terms of a nonstandard complexity metric. I confirm and sharpen that concern. The architecture's one generative invocation per turn is real, and the accuracy results in Table 3 are not in dispute. What is not established is the paper's broader assertion that computational complexity is constant in the ontology size. First, the output softmax is over E, whose size includes BERT vocabulary plus a static embedding for every domain/slot from the training ontology (Section 3.1), so the per-step arithmetic cost is not independent of the number of predefined slots. Second, the decoder is autoregressive, so the number of steps equals the generated belief-state length; in the worst case this length is proportional to the number of slots that can appear in a turn, which is bounded by n. Thus a worst-case analysis gives a term linear in n, and the empirical section cannot refute this because its theoretical multiplier K already assumes the O(1) class and because the two datasets differ along several confounded dimensions. None of this undermines the model's competitive accuracy or the value of formulating DST as hierarchical sequence generation; it means the scalability claim needs a precise restatement and a direct measurement of per-step and per-sequence costs before it can be called constant-time. Since the reader's conditional verdict already calls for that restatement, my stress-test does not move the verdict.","tokens_in":10299,"tokens_out":9804,"duration_ms":110760,"concrete_test":"Instrument the released code to log, per turn, the number of decoder calls, the total number of decoder steps, and wall-clock latency on MultiWOZ. Then run two controlled variations: (a) hold the dialogue and decoding path fixed while enlarging Es in the output matrix E with 10, 100, and 1000 zero-initialized dummy slot embeddings, verifying that the argmax over the original vocabulary is unchanged; if per-turn latency or FLOPs grow with |Es|, the O(1) claim holds only for the number of inference calls, not for compute. (b) Group test-set turns by the number of active slots in the label (e.g., 1, 2, 4, 8); if total decoder steps and latency grow with active-slot count, the per-turn cost is not constant in belief complexity and the worst-case dependence on n should be acknowledged.","verdict_should_be":"UNCHANGED","load_bearing_attack":"COMER's central novelty is the claim of O(1) inference time with respect to the number of pre-defined domains, slots, and values (Abstract; Section 3; Table 1). The paper defines ITC as the number of times inference is performed per turn, and under that definition COMER indeed makes one generative call per turn. But the Abstract and Conclusion claim 'computational complexity' is constant, which is a much stronger statement. Under a conventional operation count, the CMR decoder (Section 3.2) executes one forward pass per generated token and each pass includes a softmax over the full static embedding matrix E; Section 3.1 builds E as the concatenation of the BERT vocabulary with Es, which contains an embedding for every domain/slot seen in training. Hence per-step cost grows with the number of predefined domain/slot labels stored in Es, and the number of steps per turn is the generated belief-sequence length, which can grow with the number of slots active in the turn and, in the worst case, with n. The empirical confirmation in Section 4.3 does not settle the question: the theoretical multiplier K is computed by already assuming O(1) for n and m, and the two datasets differ simultaneously in ontology size, turn length, token counts, and validation-set size. The observed 2.54x multiplier is a consistency check, not an isolation of the dependence on n. The claim should be restated as 'one decoder invocation per turn', with the per-step softmax size and total generated-token length reported separately.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes COMER, a hierarchical sequence-generation formulation of dialogue state tracking in which a shared Conditional Memory Relation decoder first generates the sequence of domains, then slots, then values for each turn. The authors claim that this removes the dependence of inference time on the pre-defined ontology and that COMER therefore has O(1) inference-time complexity with respect to the number of domains, slots, and values. Experiments on WoZ2.0 and MultiWOZ report joint-goal accuracies of 88.6% and 48.79%, respectively, the latter being a marginal improvement over TRADE, together with ablations of the attention hierarchy, MLP, parameter sharing, ordering, nested representation, and gradient blocking. The code is released publicly.","tokens_in":10662,"tokens_out":2838,"duration_ms":33357,"significance":"If the constant-complexity claim were accurate, COMER would be the first DST with ontology-independent inference cost while matching state-of-the-art accuracy, which would be a meaningful contribution to scalable task-oriented dialogue. The paper also makes useful empirical contributions: it provides a released implementation, a careful multi-level ablation on MultiWOZ (joint domain, domain-slot, and joint goal accuracy), and a sequence-generation view of belief states that is natural for nested domain-slot-value structures. The SOTA-level joint goal accuracy of 48.79% on MultiWOZ is itself plausible and valuable, and the qualitative attention analysis is informative. However, the central scalability claim is substantially overstated under a conventional definition of computational complexity, and the reported empirical confirmation in Section 4.3 does not isolate the dependence on ontology size.","major_comments":[{"comment":"The O(1) inference-time claim is only valid under the narrow definition of 'how many times inference must be performed' in Table 1, i.e., one decoder invocation per turn. The Abstract and Conclusion instead claim constant computational complexity, which is not supported. Per generated token, the decoder computes ps = softmax(E^T h_o) over the full static embedding matrix E, and Section 3.1 defines E as the concatenation of the BERT vocabulary Ev with the extra slot embedding Es, which contains one embedding for every domain/slot seen in training. Thus the per-step softmax cost grows with the number of pre-defined labels. In addition, the number of generated tokens per turn is the length of the belief-state sequence, which can grow with the number of active slots and, in the worst case, with n. The paper should either explicitly restrict the claim to 'one decoder call per turn' and qualify all downstream statements, or provide a per-step complexity analysis that includes the output-vocabulary dimension and the autoregressive sequence length.","section":"Section 3.2 and Abstract/Conclusion"},{"comment":"The empirical confirmation of O(1) complexity is not convincing because the theoretical multiplier K is computed under the assumption h(n)=h(m)=1, i.e., the O(1) class is already assumed when defining K. The observed 2.54x multiplier is obtained by comparing two datasets that differ simultaneously in the number of slots (3 vs 35), number of values (99 vs 4510), average turns per dialogue, average tokens per turn, and validation-set size, so the comparison does not isolate the dependence on n or m. The match between 2.54 and 2.15 is a consistency check of the overall runtime model, not a test of O(1) with respect to ontology size. To support the scalability claim, the authors should report a controlled experiment (for example, varying the number of pre-defined slots on a fixed test set while measuring wall-clock time) or at least report per-token decoding time and the distribution of generated belief-state lengths on MultiWOZ.","section":"Section 4.3"}],"minor_comments":[{"comment":"The caption sentence 'The ITC is calculated based on how many times inference must be performed to complete a prediction of the belief state in a dialogue turn' is not a complete sentence; it should be integrated into a full sentence describing the definition of ITC.","section":"Table 1 caption"},{"comment":"The statement that 'the model parameters are independent of the vocabulary size' is correct but should not be conflated with computational independence: the softmax operation in ps = softmax(E^T h_o) still iterates over the full output vocabulary, including the Es component.","section":"Section 3.2"},{"comment":"The label 'Hierachical-Attn' contains a typo and should read 'Hierarchical-Attn'.","section":"Table 4"},{"comment":"The phrase 'roughly of the same magnitude as the theoretical value of 2.15' should be stated with a numerical margin or confidence interval; the reported 2.54 is 18% larger than 2.15, and without additional analysis it is unclear whether this difference is expected under the model's assumptions.","section":"Section 4.3"},{"comment":"The sentence 'The effectiveness of our hierarchical attention design is proved by an accuracy drop of 1.95%' should be softened to 'is supported by', since a single ablation is suggestive rather than a proof.","section":"Section 4.4"}],"recommendation":"major_revision","confidential_remarks":"The core empirical result, 48.79% joint goal accuracy on MultiWOZ with released code, is likely sound and of interest to the DST community. My main concern is the framing of the scalability contribution: the paper presents a meaningful architectural contribution (one generative call per turn instead of one call per slot) but overstates it as constant computational complexity. I believe this is fixable by rewriting the complexity claim and adding a controlled experiment, so I recommend major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper is worth your time for the architecture: generating the belief state as a hierarchical sequence of domains, slots, and values with shared decoders is a real departure from the scoring-over-ontology and per-slot generation approaches in the cited literature. The ablations on MultiWOZ are thorough and honest, and the code is released, which makes the empirical work reproducible. Accuracy is competitive: 48.79% joint goal on MultiWOZ, a hair above TRADE, and within noise on WoZ2.0.\n\nThe soft spot is the scalability claim. The abstract and conclusion say computational complexity is constant in the number of pre-defined slots and values. That is only true if you adopt the paper's Table 1 definition of inference time complexity as \"how many times inference is performed per turn.\" Under that count, yes, COMER is O(1). But under any conventional operation count, the decoder's per-step softmax is over the full output vocabulary E, which concatenates BERT's vocabulary with an embedding for every domain/slot seen in training, so per-step cost grows with ontology size. And the number of autoregressive steps per turn grows with the number of generated domains, slots, and values, which in the worst case is O(n). So the constant-time claim is definition-dependent, and the paper should say \"one generative call per turn\" rather than \"constant computational complexity.\"\n\nThe empirical confirmation in Section 4.3 does not fix this. The theoretical multiplier K is computed by assuming O(1) for n and m, and the two datasets differ in turn length, token counts, validation size, and ontology size simultaneously. The observed 2.54x multiplier is a consistency check, not an isolation of the dependence on n. To actually test scalability you would need to hold other factors fixed and vary ontology size.\n\nThis is not a fatal flaw. The core contribution — hierarchical sequence generation for DST — stands on its own, and the accuracy results are real. But the overreach is load-bearing for the paper's stated motivation, so it needs to be corrected before publication. The authors should also report the output vocabulary size and average generated sequence length, and ideally add a controlled scaling experiment.\n\nWho benefits: researchers working on task-oriented dialogue systems, particularly on scaling DST to many domains. The letter should go to a serious referee; the issues are fixable with a major revision that reworks the complexity analysis and tempers the claims.","headline":"A solid DST paper with a genuinely new hierarchical generation idea, but the central O(1) complexity claim is true only under a narrow definition and the empirical confirmation is weaker than presented.","tokens_in":11137,"tokens_out":2824,"would_cite":true,"duration_ms":31731,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A dialogue state tracker that generates belief states in constant time, independent of ontology size, reaches state-of-the-art multi-domain accuracy.","keywords":["dialogue state tracking","sequence generation","hierarchical decoder","multi-domain dialogue","inference complexity","joint goal accuracy","ontology-free tracking"],"falsifier":"Measure the model's wall-clock inference time on dialogues with the same number of active slots but with increasingly large predefined ontologies (e.g., adding thousands of unused values); if runtime grows linearly with the ontology size or with the generated sequence length, the O(1) complexity claim, taken as compute, is false.","tokens_in":10140,"feed_emoji":"🗣️","tokens_out":2634,"duration_ms":154578,"temperature":0.7,"pith_summary":"This paper argues that dialogue state tracking, traditionally treated as a set of per-slot classification problems over a fixed ontology, can instead be cast as hierarchical sequence generation. The proposed model, COMER, directly generates the domain, slot, and value tokens of a belief state with a shared decoder, so the number of inference passes per turn does not grow with the number of predefined slots and values. On the multi-domain MultiWOZ dataset it reaches a joint goal accuracy of 48.79%, slightly above the previous state of the art, while also matching prior accuracy on the single-domain WoZ2.0 benchmark. A sympathetic reader would take the core message to be that generation-based tracking can be both scalable and accurate, removing a practical bottleneck for dialogue systems in new or evolving domains.","feed_headline":"Constant-time dialogue state tracking reaches 48.79% on MultiWOZ","feed_subtitle":"Generating belief states hierarchically skips the ontology list, making multi-domain tracking both faster and competitive.","key_machinery":"The Conditional Memory Relation Decoder (CMRD) is the central mechanism. It is a recurrent decoder that, at each step, takes a condition vector (the parent representation from the hierarchy), attends to encoded belief-state, system, and user memories in sequence, concatenates the attention outputs into a working memory, and passes that through a multi-layer perceptron for relation reasoning. The same CMRD is reused for domain, slot, and value generation, which is what makes the hierarchy depth flexible and the parameter count independent of ontology size. The model also uses pretrained transformer embeddings and byte-pair encoding so that unseen words can be represented and generated.","core_discovery":"The central claim is that dialogue state tracking can be solved as a sequence-to-sequence problem without enumerating ontology candidates. COMER encodes the user utterance, previous system actions, and previous belief state, then uses three hierarchically stacked decoders that share all parameters: the first generates the sequence of domains, the second generates the slots for each generated domain, and the third generates the value for each generated slot. Because the model generates rather than scores domain-slot-value triples, its inference time complexity is O(1) with respect to the number of predefined domains, slots, and values. Empirically, the paper shows that the wall-clock multiplier from WoZ2.0 to MultiWOZ is close to the theoretical multiplier predicted for an O(1) model, and that the joint goal accuracy on MultiWOZ, 48.79%, is state of the art at publication time.","pith_inferences":["If the O(1) claim is interpreted as counting decoder passes, the practical compute still scales with the vocabulary size of the final softmax and with the number of generated tokens; a stricter test would measure end-to-end latency against ontology size while holding dialogue content fixed.","The same hierarchical generation idea could transfer to other structured prediction problems, such as nested information extraction or hierarchical multi-label classification, where the output has a parent-child relation.","The paper's slot-prediction weakness (58.43% slot accuracy given the correct domain) suggests that richer dialogue-history representations, rather than larger candidate sets, are the next likely source of accuracy gains.","A testable extension is to replace the frequency-based ordering of domains and slots with an adaptive or learned ordering, which could affect both accuracy and the length of generated sequences."],"forward_implications":["Dialogue systems can be extended to new domains without rebuilding a candidate list or re-scoring thousands of slot-value pairs at each turn.","The joint goal accuracy on MultiWOZ indicates that generation-based trackers can match or exceed classification-based trackers on multi-domain benchmarks.","The parameter-sharing across hierarchical levels suggests the model can in principle handle deeper or more complex belief structures, such as values with multiple alternatives, without architectural changes.","The reported 2.54x actual runtime multiplier when moving from WoZ2.0 to the five-times-larger MultiWOZ is consistent with the paper's claim of constant inference passes per turn.","Generation-based tracking can produce structured belief states that are not constrained by a predefined ontology, which may help downstream modules in open-vocabulary dialogue settings."],"supporting_citations":[{"why":"Supplies the MultiWOZ multi-domain dataset used for the main evaluation and the official baseline accuracy.","marker":"(Budzianowski et al., 2018)"},{"why":"Provides the TRADE baseline, which the paper compares against on MultiWOZ and which motivates its discussion of multi-turn mapping.","marker":"(Wu et al., 2019)"},{"why":"Supplies the NBT-CNN and baseline results on WoZ2.0 used for comparison and the O(mn) complexity reference.","marker":"(Mrksic et al., 2017)"},{"why":"Provides the pretrained transformer embeddings used for contextual and static word representations.","marker":"(Devlin et al., 2018)"},{"why":"Supplies byte-pair encoding, which the model uses to represent unseen words.","marker":"(Sennrich et al., 2016)"},{"why":"Motivates the relation-network-style MLP reasoning in the decoder.","marker":"(Santoro et al., 2017)"},{"why":"Motivates the memory-attention mechanism over encoded sources in the decoder.","marker":"(Sukhbaatar et al., 2015)"},{"why":"Supplies the StateNet baseline and the O(n) complexity comparison on WoZ2.0.","marker":"(Ren et al., 2018)"}],"fun_headline_variants":["Hierarchical generation makes dialogue state tracking ontology-free","DST without ontologies: hierarchical sequence generation scales and wins","Ontology-free DST achieves state of the art on MultiWOZ","Generating DST belief states hierarchically cuts ontology cost to zero","Hierarchical generation: constant-time DST that beats the ontology baseline"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The constant-time claim counts the number of inference passes per turn, not the actual compute, which still includes a softmax over the full predefined domain and slot vocabulary and a generation length that can grow with the number of active slots.","fun_headline_variants_meta":{"raw":{"variants":["Hierarchical generation makes dialogue state tracking ontology-free","DST without ontologies: hierarchical sequence generation scales and wins","Ontology-free DST achieves state of the art on MultiWOZ","Generating DST belief states hierarchically cuts ontology cost to zero","Hierarchical generation: constant-time DST that beats the ontology baseline"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000775,"raw_usage":{"total_tokens":3395,"prompt_tokens":880,"completion_tokens":2515,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":496,"completion_tokens_details":{"reasoning_tokens":2429}},"tokens_in":496,"tokens_out":2515,"duration_ms":15309,"temperature":1.0,"reasoning_tokens":2429,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T05:36:34.389376+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Measure the model's wall-clock inference time on dialogues with the same number of active slots but with increasingly large predefined ontologies (e.g., adding thousands of unused values); if runtime grows linearly with the ontology size or with the generated sequence length, the O(1) complexity claim, taken as compute, is false.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Motivates the relation-network-style MLP reasoning in the decoder."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Motivates the memory-attention mechanism over encoded sources in the decoder."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the StateNet baseline and the O(n) complexity comparison on WoZ2.0."}],"review_version":1}