{"id":"c35cc457-69ae-485d-88ba-3d88390d7ebb","arxiv_id":"1908.07705","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A copy-enhanced multi-encoder-decoder model for dialogue state tracking that generates unknown slot values by copying from both the dialogue and the ontology.","lead":"This paper proposes CEDST, a dialogue state tracker that generates slot values by copying words from the conversation or from a pre-defined ontology. It reports state-of-the-art results on two public datasets and on new datasets built by hiding some slot values.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The unknown-value evaluation masks closed-set WoZ2 values and only tests verbatim copying; it does not establish generation of normalized forms of genuinely novel values.","rationale":"The reader identified the faithfulness of the constructed 'unknown value' datasets as the weakest assumption, and I agree that random masking is the key evidential weak point. My pass sharpens the concern by tying it to the model's output mechanism: because CEDST's copy mode emits verbatim spans and its generation mode is restricted to known ontology values, the evaluation can only succeed when an unseen value is mentioned in exactly its canonical surface form. This means the constructed datasets do not merely risk being unrepresentative; they are structurally incapable of testing lexical normalization of novel values, which is one of the original motivations stated in the introduction. The bAbI OOV result provides some independent support for the verbatim-copy capability, but it is a simple setting and does not close the gap. I do not see an internal inconsistency in the equations, and the WoZ2 joint-goal result is plausible, so the paper still merits conditional acceptance with the condition that unknown-value claims be demonstrated on a true held-out value split with paraphrases and released data. This does not move the reader's verdict, hence UNCHANGED.","tokens_in":951,"tokens_out":1008,"duration_ms":119715,"concrete_test":"Build a held-out-value split of WoZ2 instead of a random mask: partition each slot's values into train and test sets, remove all train dialogues containing test values, remove test values from the known-value memory K, and in the test set replace half of the mentions with paraphrase surface forms (e.g., 'mid-priced' for 'moderate', 'inexpensive' for 'cheap', 'downtown' for 'centre') while keeping the original canonical labels. Run CEDST with released code and report joint-goal accuracy separately for verbatim novel values and paraphrased novel values. If paraphrased novel values are near chance or substantially below verbatim values, the 'all possible values' claim fails for lexical variants, and the constructed-UNK numbers cannot be cited as evidence of open-domain OOV handling.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central capability claim rests on the constructed UNK datasets in Section 4.1, where 'unknown' values are randomly selected from the closed WoZ2 ontology and omitted from training. This tests extraction of values that are merely unseen in training, not generation of out-of-ontology values in general. The architecture sharpens the gap: in Section 3.4, Eq. (19)-(20) define the output distribution over memory entries, so the copy mode can only emit a contiguous dialogue span verbatim. For a known value the ontology path can supply the canonical string, but for a novel value there is no mechanism to map a surface variant such as 'mid-priced' to the canonical 'moderate'. Thus the model succeeds on an unknown value only when the text happens to contain the exact normalized string. Masking values from a closed set also preserves the lexical and embedding statistics of those values, so the reported correct-UNK ratios in Section 4.4 are upper bounds for realistic open-vocabulary OOV, where values may be multiword, paraphrased, or absent from the pretrained embeddings. The bAbI OOV result is cleaner but only demonstrates verbatim direct-mention copying on a simple task and does not repair the normalization limitation. The reader's concern about the constructed datasets is therefore not just about dataset fidelity; it is the load-bearing point on which the 'all possible values' claim depends.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes CEDST, a sequence-to-sequence dialogue state tracking model that combines a copy mechanism with multiple private/shared encoders and decoders. The model decomposes the state space into slot-specific memories formed from the user utterance, system response/actions, and the ontology, and it generates each slot value either by selecting a known ontology value or by copying a span from the dialogue. The authors evaluate on WoZ2, bAbI Task 5, and WoZ2-based datasets constructed by masking 20%, 40%, or 60% of known values from the training ontology. They report joint-goal accuracy of 89.6% on WoZ2, perfect or near-perfect results on bAbI for p=0, and small improvements over GLAD on the constructed datasets, and conclude that CEDST can generate all possible values including unknown values.","tokens_in":8700,"tokens_out":7346,"duration_ms":114807,"significance":"If the central claim holds, the paper offers a practical generative DST that removes the need for hand-crafted normalization lists and handles values outside a fixed ontology, a real bottleneck for scalable dialogue systems. The multi-encoder-decoder decomposition is a reasonable way to reduce a large state space, and the bAbI out-of-vocabulary results show that the copy mechanism can extract previously unseen values when they are directly mentioned in the dialogue. The paper is also transparent about the architecture and includes ablations. However, the main evidence for the 'all possible values' claim rests on constructed datasets in which 'unknown' values are masked closed-set values, and the copy mechanism can only emit verbatim spans from the input; the results therefore demonstrate a narrower capability than the abstract claims, and the experimental support is thin: single-run numbers, small margins over GLAD, and an incomplete bAbI table.","major_comments":[{"comment":"The central claim that CEDST 'can effectively generate all possible values including unknown values' is not supported for genuinely out-of-ontology values. In the copy mode, the output distribution is defined over the memory entries M, so the model can only emit a contiguous span that appears verbatim in the dialogue or a value from the ontology; there is no mechanism to map a surface variant such as 'moderately' to the canonical 'moderate' when that canonical string is absent from both the dialogue and the ontology. In the constructed datasets, 'unknown' values are randomly selected known WoZ2 values whose lexical and embedding statistics remain in-distribution, and the reported correct-UNK ratios in Figure 4 are therefore upper bounds for realistic open-vocabulary values that may be multiword, paraphrased, or partially covered by GloVe. I recommend either evaluating on truly novel values with canonicalization requirements or substantially tempering the generalization claim.","section":"Section 3.4, Eqs. (19)-(20); Section 4.1"},{"comment":"CEDST results are reported as single runs without error bars, while the strongest baseline, GLAD, is reported as 88.1±0.4 on WoZ2 joint goal; the 1.5-point margin for CEDST is within a plausible noise range, so the state-of-the-art claim is not statistically established. In Table 2, the p=0.1 columns for CEDST on both Food and Location are missing (shown as dashes), even though the text claims CEDST outperforms PtrNet; this makes the comparison incomplete and prevents verification of the claim for that setting.","section":"Section 4.3, Tables 1 and 2"},{"comment":"The gains over GLAD on the constructed datasets are 1.7, 3.2, and 0.2 percentage points for the 20%, 40%, and 60% UNK conditions, respectively, with no significance or variance information. At 60% UNK, CEDST is essentially at the floor (11.2% vs. 11.0% for GLAD). The conclusion that CEDST achieves state-of-the-art performance on the constructed datasets is therefore not strongly supported by the reported numbers.","section":"Section 4.4, Table 3"},{"comment":"The '- copy' ablation removes the copy mechanism and the utterance hidden from the memory M simultaneously, so it cannot isolate the contribution of the copy mechanism. The observed degradation could be due to removing the utterance memory rather than to the copy mechanism itself, weakening the stated conclusion that 'copy mechanism generates unknown values effectively.'","section":"Section 4.5, '- copy' ablation"}],"minor_comments":[{"comment":"The text refers to 'Table 2' when presenting the constructed-dataset results, but the actual table is numbered Table 3; the cross-reference should be corrected.","section":"Section 4.4"},{"comment":"The p=0.1 columns for CEDST should be completed or an explicit explanation should be given for their absence.","section":"Table 2"},{"comment":"The paper uses 'Copy-Enhanced' in the title and abstract but 'copy-augmented' in the body; the terminology should be made consistent.","section":"Title and Section 3"},{"comment":"The phrase 'Selected values are seamed as unknown values' contains a typo; it should presumably read 'treated as unknown values'.","section":"Section 4.1"},{"comment":"The word 'fratures' in the first paragraph is a typo for 'features'.","section":"Section 2.1"},{"comment":"Figure 2 contains an unexplained Fourier-series expression for P(y_t|M_a); this looks like a leftover placeholder and should be removed or properly explained.","section":"Figure 2"},{"comment":"For the multi-value request slot, the paper says predicted values can be obtained with 'some simple segmentation rules' but never specifies these rules; because they affect the reported request accuracy, they should be described.","section":"Section 3.4"},{"comment":"The description 'API calls in Task 5 of the bAbI dataset is regarded as the state' is grammatically unclear and should be rewritten to specify how dialogue states are derived from the API calls.","section":"Section 4.1"},{"comment":"The 'simple rule' for integrating turn goals into joint goals is a hand-crafted component; the paper should state whether the reported results are sensitive to this rule, especially since it is not part of the multi-decoder.","section":"Section 4.3"},{"comment":"The definition of 'generated unknown values' in Figure 4 is not precise; it should state whether these counts are per-slot predictions, joint states, or something else.","section":"Figure 4"}],"recommendation":"major_revision","confidential_remarks":"The stress-test concern is on point: the constructed UNK datasets are the load-bearing evidence for the paper's headline claim, and the copy mechanism's verbatim-only mode is a structural limitation for normalization of genuinely novel values. The bAbI OOV results are a useful positive result but should not be overgeneralized. I would ask the authors for a revised evaluation with truly out-of-distribution values, statistical reliability measures, and a completed bAbI table before publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The multi-encoder-decoder with copy mechanism is a reasonable engineering contribution, but the paper overstates what it proves about unknown values. The constructed UNK datasets mask values from the closed WoZ2 ontology and treat them as unseen; the copy mechanism (Eq. 19-20) can only emit contiguous dialogue spans verbatim, so the model succeeds on an unknown value only when the text happens to contain the exact normalized string. That is not the same as generating canonical forms for genuinely new surface variants like 'mid-priced' → 'moderate'. The stress-test note gets this right, and it is the load-bearing point on which the 'all possible values' claim depends.\n\nWhat is actually new here is the combination of GLAD-style private/shared encoders with a copy-augmented multi-decoder for DST, plus the constructed datasets for evaluating unseen values. The paper is clearly written and the ablation study is informative: each component (multi-encoder, multi-decoder, copy, self-attention) contributes, and the shared LSTM matters less than the paper implies. The reported WoZ2 numbers are plausible and the bAbI OOV results, though only for verbatim copying, are a legitimate demonstration on a simpler task.\n\nSoft spots, in rough proportion: the unknown-value evaluation is the main weakness. Masking values from a closed set preserves their lexical and embedding statistics, so the reported correct-UNK ratios are upper bounds for realistic OOV, where values may be multiword, paraphrased, or absent from the pretrained vocabulary. The bAbI table is also incomplete—no p=0.1 columns for CEDST, which is odd given the headline claim. There are no error bars on the WoZ2 results, and the margins over GLAD (1.5% joint goal) are small. The constructed datasets are not released, which makes it hard to verify the comparison. These are not fatal, but they justify caution.\n\nWho gets value from this paper? Researchers working on DST or other slot-filling tasks, especially those interested in copy mechanisms and multi-space encoders. It deserves a serious referee because the architecture is sound and the task is important, but the authors should temper the 'all possible values' claim and add experiments with genuinely novel surface forms—paraphrases, values that never appear in the ontology, or words held out from the embeddings. For peer review, I would recommend major revision rather than acceptance as-is.","headline":"A plausible DST architecture with a copy mechanism, but the unknown-value evaluation only tests verbatim copying of closed-set values, so the central claim is overstated.","tokens_in":9169,"tokens_out":2710,"would_cite":false,"duration_ms":61198,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"CEDST generates all dialogue-state values, including values absent from the ontology, by copying spans from the conversation and reading ontology-aware slot-specific memories, and it reports state-of-the-art joint-goal accuracy on WoZ2…","keywords":["dialogue state tracking","copy mechanism","unknown slot values","multi-encoder-decoder","task-oriented dialogue","heterogeneous texts","ontology","out-of-vocabulary values"],"falsifier":"A test set of genuinely novel slot values, such as restaurant names, cuisine words, or area names never appearing in training, would settle it: if CEDST's accuracy on those values stays near its masked-value results, the unknown-value claim is sound; if it drops sharply, the masking protocol overestimates generalization.","tokens_in":8236,"feed_emoji":"🗣️","tokens_out":6473,"duration_ms":486907,"temperature":0.7,"pith_summary":"The paper argues that dialogue state tracking should be treated as a generative problem over heterogeneous texts: the slot value can come from either the dialogue context or the pre-defined ontology. It claims that a copy mechanism lets a tracker emit values unseen in the ontology by copying spans from the conversation, while encoding known ontology values into the same vector spaces prevents the semantic loss that hurt earlier generative models. The proposed CEDST model splits the large state space into per-slot private and shared encoder-decoder spaces, and reports state-of-the-art joint-goal accuracy on WoZ2 and bAbI, plus gains on artificially constructed datasets where known values are masked as unknown. A reader should care because going beyond a fixed ontology is what makes dialogue state tracking usable in dynamic real-world domains with new restaurants, new movies, or unbounded values.","feed_headline":"Dialogue tracker copies unseen values to state-of-the-art","feed_subtitle":"CEDST hits 89.6% joint-goal accuracy on WoZ2 and 100% on bAbI's out-of-vocabulary test.","key_machinery":"The mechanism is a copy-augmented multiple encoder-decoder. A context multi-encoder runs a private and a shared BiLSTM with self-attention for each slot to produce per-slot memories $M_a$, $M_f$, $M_p$, and $M_r$; a known-value multi-encoder maps ontology values and special values such as 'none' and 'don't care' into the same slot-specific spaces. The multi-decoder combines private and shared LSTM decoders and, for each slot, computes a distribution over copying a word from the dialogue and generating a value from ontology memory; for multi-value slots it selects all candidates above a threshold in one step. This carries the argument by letting known and unknown values be produced from one unified distribution, with no hand-crafted normalization list.","core_discovery":"The central claim is that all possible slot values, including values never seen in training, can be generated by copying words from the dialogue context or generating them from ontology-aware memory. CEDST does this with multiple encoders that map dialogue and known ontology values into one space per slot, and multiple decoders that read those reduced spaces; the copy mechanism decides at the first decoding step whether to emit a known value from the ontology or copy an unknown value word-by-word from the utterance. In experiments, CEDST achieves 89.6% joint-goal accuracy on WoZ2 and 100% on both food and location slots in bAbI's out-of-vocabulary test, outperforming the pointer-network baseline, and improves joint-goal accuracy on constructed datasets with 20%, 40%, and 60% masked unknown values.","pith_inferences":["The paper's unknown-value evaluation masks known WoZ2 values; a stricter test would use values whose surface forms never appear anywhere in training, and performance on truly novel forms may be lower than the 20–60% masking results suggest.","The same copy-plus-multi-encoder design could transfer to other tasks with open vocabularies and heterogeneous inputs, such as slot filling for natural-language understanding, where values like dates or song titles must be extracted from context rather than selected from a fixed set.","Combining the copy mechanism with a discriminative candidate-set scorer could yield a hybrid that keeps the generative model's flexibility while letting a reranker validate copied spans against dialogue context, potentially improving precision on copied unknown values."],"forward_implications":["State tracking no longer needs an exhaustive candidate ontology or a hand-written normalization list, because values can be copied in normalized form from dialogue or ontology.","Decomposing the state space into per-slot private and shared representations improves joint-goal accuracy on WoZ2 over the shared-only architecture (89.6% versus 85.8% when the multi-encoder is removed).","Copying from dialogue context is what enables unknown values; removing the copy mechanism and utterance hidden degrades joint-goal accuracy on the 40% unknown-value dataset from 32.4% to 29.4%.","The approach handles both single-value and multi-value slots within one architecture, selecting all values above a threshold for request slots.","On bAbI's out-of-vocabulary test, copying from the utterance reaches 100% food and location accuracy, matching or beating a pointer network trained with targeted feature dropout."],"supporting_citations":[{"why":"Supplies the copy mechanism that lets CEDST copy words from the dialogue as unknown values.","marker":"[4]"},{"why":"The pointer-network generative baseline for unknown slot values; CEDST is compared against it on bAbI and avoids its hand-crafted normalization list.","marker":"[17]"},{"why":"Provides the private-and-shared multi-encoder design CEDST adapts and a strong baseline on WoZ2.","marker":"[19]"},{"why":"Defines the neural belief tracking setup and baselines on WoZ2 that CEDST must outperform.","marker":"[10]"},{"why":"Provides the WoZ2 restaurant-finding dialogue dataset and the delexicalised dictionary baseline.","marker":"[16]"},{"why":"Provides bAbI Task 5 with an out-of-vocabulary test set used to measure unknown-value generation.","marker":"[2]"},{"why":"Prior work on handling dynamic and unbounded value sets in dialogue state tracking, giving context for why unknown values matter.","marker":"[13]"}],"fun_headline_variants":["Copying dialogue text tracks unseen slot values","Multi-encoder DST copies unknown values from context","CEDST copies unknown values, hits new accuracy","Dialogue state tracker uses copy mechanism for unknowns"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that randomly masking known WoZ2 values and treating them as 'unknown' replicates how genuinely new out-of-ontology values behave, so the measured copy performance is evidence of real generalization rather than an artifact of masked vocabulary.","fun_headline_variants_meta":{"raw":{"variants":["Copying dialogue text tracks unseen slot values","Multi-encoder DST copies unknown values from context","CEDST copies unknown values, hits new accuracy","Dialogue state tracker uses copy mechanism for unknowns"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000575,"raw_usage":{"total_tokens":2715,"prompt_tokens":949,"completion_tokens":1766,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":565,"completion_tokens_details":{"reasoning_tokens":1707}},"tokens_in":565,"tokens_out":1766,"duration_ms":186439,"temperature":1.0,"reasoning_tokens":1707,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T11:58:25.733569+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A test set of genuinely novel slot values, such as restaurant names, cuisine words, or area names never appearing in training, would settle it: if CEDST's accuracy on those values stays near its masked-value results, the unknown-value claim is sound; if it drops sharply, the masking protocol overestimates generalization.","supporting_citations":[{"cited_title":"In: Proceedings of ACL","cited_arxiv_id":null,"evidence_quote":"Supplies the copy mechanism that lets CEDST copy words from the dialogue as unknown values."},{"cited_title":"In: Proceedings of ACL","cited_arxiv_id":null,"evidence_quote":"The pointer-network generative baseline for unknown slot values; CEDST is compared against it on bAbI and avoids its hand-crafted normalization list."},{"cited_title":"In: Proceedings of ACL","cited_arxiv_id":null,"evidence_quote":"Provides the private-and-shared multi-encoder design CEDST adapts and a strong baseline on WoZ2."},{"cited_title":"In: Proceedings of ACL","cited_arxiv_id":null,"evidence_quote":"Defines the neural belief tracking setup and baselines on WoZ2 that CEDST must outperform."},{"cited_title":"In: Proceedings of EACL","cited_arxiv_id":null,"evidence_quote":"Provides the WoZ2 restaurant-finding dialogue dataset and the delexicalised dictionary baseline."},{"cited_title":"CoRR (2016)","cited_arxiv_id":null,"evidence_quote":"Provides bAbI Task 5 with an out-of-vocabulary test set used to measure unknown-value generation."}],"review_version":1}