{"id":"5c6da27f-8f45-48a7-96ae-a781679bf358","arxiv_id":"2412.03270","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":1,"one_line_summary":"IDIC-DST improves few-shot dialogue state tracking by extracting user intent to augment dialogue information and retrieve in-context examples, achieving reported SOTA on MultiWOZ 2.1 and 2.4.","lead":"This paper introduces IDIC-DST, a method for few-shot dialogue state tracking that extracts user intent with a fine-tuned T5 model and uses it to retrieve examples and guide a code-generating LLM. The authors report state-of-the-art joint goal accuracy on MultiWOZ 2.1 and 2.4 at a 1% few-shot setting.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table II reports 52.66 JGA for IDIC-DST on MultiWOZ 2.4 at 1% few-shot, while Table IV reports the full method (DCA+IER) at 43.39 on the same setting, an unexplained 9.27-point gap. This internal contradiction makes the SOTA claim unverifiable as written.","rationale":"I read the paper in good faith: the proposed modules are clearly described and the ablation gains are large, so the method itself may be salvageable. My concern is not about the idea but about the evidence for the headline SOTA claim. The strongest claim is the 43.11 / 52.66 JGA in Table II, and to accept it one must be able to reproduce the reported configuration. Table IV, however, reports the full method at 43.39 on the same 1% MultiWOZ 2.4 setting, a 9.27-point shortfall from Table II. There is no explanation for this gap, and Section III-B does not specify which 1% of the dataset was used for training the T5 and SBERT models and for the retrieval pool. The reader's weakest_assumption focused on possible test-set leakage; I partially agree, but the Table II/Table IV discrepancy is more decisive because it is an observable internal contradiction rather than a speculation about split composition. The unresolved mismatch means the paper contains two incompatible accounts of the system's performance, so the central claim is not verifiable from the text. I would keep the reader's REJECT: the authors should be asked to reconcile the two numbers and disclose the exact 1% split before the SOTA claim can be evaluated.","tokens_in":8483,"tokens_out":7582,"duration_ms":66032,"concrete_test":"Request the released code and data-split file, then rerun the 1% MultiWOZ 2.4 configuration from Table II exactly as specified in Section III-B and the ablation harness of Table IV. If the reproduced JGA matches 43.39 and not 52.66, the headline main result is not reproducible from the documented setup; if it matches 52.66, the Table IV row was labeled or configured differently and that difference must be disclosed. The same check should confirm whether the 1% sample is drawn only from the MultiWOZ training split and contains no evaluation dialogues.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing concern is the unexplained 9.27-point JGA gap between the main result and the ablation for the same method and dataset. In Table II, IDIC-DST is reported at 52.66 on 1% MultiWOZ 2.4. In Table IV, the row labeled 'w/ DCA,IER', described in Section III-D as the full method after adding both proposed modules to IC-DST, is reported at 43.39 on the same 1% setting. The paper does not state whether the two tables use the same 1% sample, the same retriever, the same k, the same prompt template, or the same evaluation split. Section III-B only says 'we extracted 1% of the MultiWOZ dataset' without specifying split, and no code or configuration is released. Because the central SOTA claim rests on Table II, and Table IV is the only place where the full system is independently described, the paper currently contains two incompatible accounts of the system's performance. A reader cannot tell which number is the true JGA, or whether the Table II number was produced by an undocumented different setting, possibly one that uses evaluation dialogues in the retrieval pool. The discrepancy therefore blocks verification of the stated claim.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes IDIC-DST, a few-shot dialogue state tracking (DST) method built on CodeLlama-7B. A fine-tuned T5-small model first extracts the user's intent and augments the dialogue information; an SBERT retriever then retrieves top-k in-context examples from a masked and rewritten version of the augmented dialogue; the LLM is prompted with these examples plus the augmented information to generate a SQL query that updates the dialogue state. The authors report state-of-the-art joint goal accuracy (JGA) on MultiWOZ 2.1 and MultiWOZ 2.4 under a 1% few-shot setting (43.11 and 52.66, respectively) and include ablation, task-oriented dialogue system, and case-study analyses.","tokens_in":8754,"tokens_out":4586,"duration_ms":43658,"significance":"If the reported results are correct, IDIC-DST would be a useful contribution: it addresses two recognized difficulties in few-shot DST, namely implicit user input and noisy dialogue history, and it does so with a relatively small generation model (7B parameters). The text-to-SQL formulation is an interesting alternative to the code-generation framing of prior in-context-learning DST work, and the case study (Table V) illustrates the intended behavior concretely. However, the manuscript as written does not provide reliable evidence for its central claim: there is an unexplained internal contradiction in the main results, the 1% data split is underspecified with a serious leakage risk, the design includes the supervised intent prediction as a reference output in the LLM prompt, and no variance or seed information is reported. These issues block verification of the state-of-the-art claim.","major_comments":[{"comment":"Table II reports IDIC-DST at 52.66 JGA on 1% MultiWOZ 2.4, while Table IV reports 43.39 JGA for the row labeled 'w/ DCA,IER', which Section III-D explicitly describes as the full method after adding both proposed modules to IC-DST. On the same 1% setting these two entries should describe the same system and should match; the 9.27-point difference is unexplained. The paper does not state whether the two tables use the same 1% sample, the same retriever, the same value of k, the same prompt template, or the same evaluation split. This internal contradiction makes the central state-of-the-art claim unverifiable as written.","section":"Table II vs. Table IV"},{"comment":"The text states 'we extracted 1% of the MultiWOZ dataset' for training the T5 NLU model, the SBERT retriever, and for use as the retrieval sample pool, but it does not specify that this 1% is drawn exclusively from the training split and excludes evaluation dialogues. Because the same 1% is used for supervised training, retrieval fine-tuning, and the retrieval pool, any overlap with test dialogues would directly inflate JGA through memorization and retrieval leakage. The authors must specify the exact split, verify no test-set contamination, and ideally report results across multiple random 1% samples.","section":"Section III-B"},{"comment":"The augmented dialogue information D'_t includes the T5-extracted intent It, and Eq. (3) shows that It remains unmasked in D''_t; Section II-C then feeds D''_t into the LLM prompt. The case study in Table V makes the mechanism explicit: 'IDIC-DST referance output: (attraction-area:south)' is provided to the LLM, and the final output is exactly that value. The LLM's final dialogue state may therefore be largely a copy of the supervised T5 prediction, so the observed gains in Table IV cannot be attributed to in-context learning or retrieval alone. The authors should report an ablation that removes It from the LLM prompt while keeping it for retrieval, and should also report the JGA of the T5 intent extractor alone as a reference point.","section":"Section II-C and Table V"},{"comment":"All JGA numbers appear to come from a single run with no variance or number of seeds reported. In a few-shot setting with only 1% of the data, results are highly sensitive to the random split, model initialization, and retrieval order; for example, Table II shows a 0.02-point difference between IDIC-DST (43.11) and IC-DST Codex (43.13) on MultiWOZ 2.1, which cannot be interpreted without error bars. The authors should report the mean and standard deviation over at least 3–5 seeds and indicate whether the differences are statistically significant.","section":"Tables II and IV"},{"comment":"The description of the retrieval module and the experimental setup omits several details needed for reproducibility: the value of k, the exact rewriting procedure for the user's input, the masking implementation, the full prompt template for CodeLlama-7B, and decoding parameters (temperature, max length, sampling strategy). Without these details, the reported experiments cannot be reproduced or independently checked.","section":"Sections II-B and III-B"}],"minor_comments":[{"comment":"The table contains typos: 'referance' should be 'reference', and 'WRONG MASSAGES' should be 'WRONG MESSAGES'.","section":"Table V"},{"comment":"The abbreviation for the Intent-driven Dialogue Information Augmentation module is inconsistent: the text uses 'IDA' while Table IV uses 'DCA' in the row labels; the table should use the same acronym as the text.","section":"Section III-D and Table IV"},{"comment":"SM2-11B is listed in both the '1–10B' and the '10–100B' parameter-size groups, with different JGA values in each row; since 11B belongs in the latter group, the duplicated row is confusing and should be removed or corrected.","section":"Table II"},{"comment":"The sentence describing the TOD system says 'the natural language understanding module employs a template-based generation method'; this appears to be a typo for 'natural language generation module', and the roles of the policy and NLG modules should be clarified.","section":"Section III-C"}],"recommendation":"major_revision","confidential_remarks":"The framework is coherent and the direction is potentially publishable, but the manuscript currently contains a direct contradiction between the headline result and the full-system ablation, plus an unresolved data-leakage risk in the 1% split. These are load-bearing issues that require additional experiments and clarifications rather than local edits; I would recommend asking the authors to address them in a revised submission."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper has a genuinely useful intuition: extract the user's intent with a small NLU model, then use that intent both to augment the current dialogue context and to retrieve better in-context examples for an LLM-based DST system. The text-to-SQL framing for state updates is a new twist, and the ablation shows each module helps on its own. The case study makes the mechanics clear, and the writing is straightforward.\n\nBut the central claim is not supported as written. The stress-test note is correct: Table II reports IDIC-DST at 52.66 JGA on 1% MultiWOZ 2.4, while Table IV reports the full method (w/ DCA,IER) at 43.39 on the same setting. That is an unexplained 9.27-point gap. The paper never states whether the two tables use the same 1% sample, same retriever, same k, or same prompt template. Since the SOTA claim rests on Table II and the ablation row is the only other account of the full system, a reader literally cannot tell which number is the true JGA. This is a load-bearing flaw, not a cosmetic one.\n\nSecond, the circularity concern is real. Table V shows that the LLM is given \"IDIC-DST reference output\"—the predicted intent, which is exactly the slot-value pair that the final state should output. The final answer then copies that reference. The method section frames intent as part of the augmented dialogue information, which could be legitimate, but the case study reveals something closer to peeking. That obscures the marginal contribution of the LLM and inflates the apparent gain from retrieval.\n\nThird, the 1% split is undefined. The paper says \"we extracted 1% of the MultiWOZ dataset\" without specifying that it comes from the training split only. If the retrieval pool or the T5/SBERT training included evaluation dialogues, the numbers would be inflated. There is also no variance reporting or multiple seeds, so we cannot assess stability.\n\nWhat is good: the module design is clean, the ablation is informative, and the practical TOD experiment (Table III) shows a real downstream benefit. The novelty is incremental but legitimate.\n\nWho is this for? Researchers working on few-shot DST would want to read it, but only after the authors resolve the discrepancy and clarify the split. As a referee, I would not reject on method—the idea is worth pursuing. But I would require a full explanation of the Table II/Table IV gap, a precise description of the data split, and a prompt template that does not feed the answer back as a reference. Code or released configs would settle most of this.\n\nI would send this to peer review: the issues are addressable, the question is relevant, and the current version is not acceptable. A serious referee could extract a solid paper from this, but only if the authors are willing to be transparent.","headline":"The intent-driven retrieval idea is sensible, but the central SOTA claim is unverifiable because the same configuration scores 52.66 in Table II and 43.39 in Table IV, and the method leaks its own predicted intent into the prompt as a reference output.","tokens_in":9290,"tokens_out":2127,"would_cite":false,"duration_ms":21646,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Intent-driven in-context learning lifts few-shot dialogue state tracking to state of the art.","keywords":["dialogue state tracking","few-shot learning","in-context learning","intent extraction","retrieval-augmented generation","text-to-SQL","MultiWOZ","task-oriented dialogue"],"falsifier":"Inspect the data split used for the 1% sample; if any evaluation dialogue appears in the training sample or the retrieval pool, re-run the experiments with a strictly training-only sample and pool. A substantial drop in the reported 43.11 and 52.66 joint goal accuracy under that clean split would falsify the claim that intent-driven in-context learning alone produces the state-of-the-art results.","tokens_in":8273,"feed_emoji":"🎯","tokens_out":6753,"duration_ms":57061,"temperature":0.7,"pith_summary":"The paper aims to show that few-shot dialogue state tracking (DST) can be made substantially more accurate by putting the user's intent at the center of the prompt construction. Its method, IDIC-DST, extracts the current intent with a small fine-tuned text-to-text model, appends it to the dialogue information, and uses it to mask noisy dialogue history and rewrite the user's utterance before retrieving similar examples. The cleaned prompt is sent to a 7-billion-parameter code-oriented language model that produces an SQL statement representing the dialogue-state update. On 1% of MultiWOZ 2.1 and 2.4, the paper reports joint goal accuracy of 43.11 and 52.66, respectively, surpassing prior few-shot DST systems including one with more than 100 billion parameters. The claim is that intent extraction plus retrieval purification, rather than a larger generator, is the key to few-shot DST.","feed_headline":"Intent-driven prompts beat 175B models on few-shot dialogue tracking","feed_subtitle":"Adding user intent to retrieval lifts joint goal accuracy to 43.11 and 52.66 on MultiWOZ with a 7B generator.","key_machinery":"The mechanism is a two-module pipeline built around the user's intent. The Intent-driven Dialogue Information Augmentation module fine-tunes a T5 model to extract the current intent and concatenates it with the dialogue context, giving the LLM explicit access to what the user actually wants. The Intent-driven Examples Retrieval module masks dialogue history, current state, and previous state from the augmented information, rewrites the user input in terms of the intent, and uses a fine-tuned Sentence-BERT retriever to find the top-k similar examples by an average F1 score over slot sets and slot-value sets. The final component casts dialogue state updating as text-to-SQL generation: a fixed SQL command initializes each domain as a table, and the LLM's generated SQL query is parsed into state changes. The intent is the load-bearing representation: it cleans the retrieval query and supplies the reference signal that lets the generator handle implicit language.","core_discovery":"The central claim is that the bottleneck in few-shot DST is the information put into the prompt, not the generative model. The authors find that examples retrieved using only the user's current intent give better guidance than examples retrieved using dialogue history or previous state, because history contains too much irrelevant content. They also find that appending the extracted intent to the dialogue context helps the LLM resolve implicit user inputs. Based on these findings, the proposed system augments the dialogue representation with the intent, masks history and prior state tokens before retrieval, rewrites the user input into an explicit form driven by the intent, and fine-tunes a sentence-embedding retriever with a slot/slot-value similarity score. The final prompt is formatted as text-to-SQL, which makes the state update easy to parse. The reported outcome is state-of-the-art joint goal accuracy on both MultiWOZ 2.1 and MultiWOZ 2.4 at 1% few-shot, with gains over the prior best system and parity or better relative to a 175-billion-parameter baseline.","pith_inferences":["Because both auxiliary modules are trained on the same 1% sample used for the few-shot evaluation, the method's 'few-shot' budget is larger than the name suggests; a comparison that charges all methods for every trained component could change the ranking.","The finding that dialogue history hurts retrieval suggests a general pre-retrieval denoising step: strip historical context and surface the underlying intent before embedding, which could transfer to other retrieval-augmented generation tasks.","The text-to-SQL formulation is a natural fit for slot-value updates and could be reused for any structured schema-tracking problem, such as form filling or API argument tracking.","If the 1% sample includes evaluation dialogues, the reported numbers would be inflated; the paper's split choice should be verified to settle the claim."],"forward_implications":["At 1% few-shot, IDIC-DST reports joint goal accuracy of 43.11 on MultiWOZ 2.1 and 52.66 on MultiWOZ 2.4, outperforming the previous best few-shot DST models in the comparison.","The method achieves this with a 7-billion-parameter generator, matching or beating a 175-billion-parameter in-context baseline, which suggests parameter scale is not the decisive factor in few-shot DST.","Adding the intent-driven dialogue information augmentation alone accounts for a large share of the gains over the base in-context learner, and adding the intent-driven retrieval on top produces the full improvement.","Reformulating state updates as SQL generation gives a structured, parseable representation that can be applied to update prior dialogue state.","In an end-to-end task-oriented dialogue system, the improved state tracker also raises booking rate, completion rate, and F1 while reducing average turns."],"supporting_citations":[{"why":"The base in-context learning framework for DST that the proposed method extends and its main comparison baseline.","marker":"[14]"},{"why":"Prior work on diverse retrieval-augmented in-context learning for DST, motivating the retrieval-based example selection.","marker":"[15]"},{"why":"The T5 text-to-text transformer used as the architecture for the fine-tuned intent extractor.","marker":"[16]"},{"why":"MultiWOZ 2.1, one of the two datasets used for few-shot evaluation.","marker":"[17]"},{"why":"MultiWOZ 2.4, the corrected dataset used as the second benchmark.","marker":"[18]"},{"why":"Joint Goal Accuracy, the evaluation metric for all main results.","marker":"[19]"},{"why":"Sentence-BERT, the retriever backbone fine-tuned for intent-driven example selection.","marker":"[20]"},{"why":"The 7-billion-parameter code-oriented language model used to generate SQL state updates.","marker":"[21]"},{"why":"The strongest prior few-shot DST baseline that IDIC-DST claims to surpass.","marker":"[26]"}],"fun_headline_variants":["Intent beats history for few-shot dialogue state tracking","Smarter retrieval: intent-driven few-shot DST hits new SOTA","Small LLM tops 175B with intent-focused DST prompts","Retrieve by intent: few-shot DST outperforms large models"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing assumption is that the 1% of MultiWOZ used to train the intent extractor and the retriever and to form the retrieval pool comes only from the training split; if evaluation dialogues are included in that 1%, the reported accuracy gains could come from retrieval leakage rather than from the method.","fun_headline_variants_meta":{"raw":{"variants":["Intent beats history for few-shot dialogue state tracking","Smarter retrieval: intent-driven few-shot DST hits new SOTA","Small LLM tops 175B with intent-focused DST prompts","Retrieve by intent: few-shot DST outperforms large models"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000185,"raw_usage":{"total_tokens":1324,"prompt_tokens":948,"completion_tokens":376,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":564,"completion_tokens_details":{"reasoning_tokens":304}},"tokens_in":564,"tokens_out":376,"duration_ms":3807,"temperature":1.0,"reasoning_tokens":304,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T22:34:49.799769+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Inspect the data split used for the 1% sample; if any evaluation dialogue appears in the training sample or the retrieval pool, re-run the experiments with a strictly training-only sample and pool. A substantial drop in the reported 43.11 and 52.66 joint goal accuracy under that clean split would falsify the claim that intent-driven in-context learning alone produces the state-of-the-art results.","supporting_citations":[{"cited_title":"Smith, and Mari Ostendorf","cited_arxiv_id":null,"evidence_quote":"The base in-context learning framework for DST that the proposed method extends and its main comparison baseline."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Prior work on diverse retrieval-augmented in-context learning for DST, motivating the retrieval-based example selection."},{"cited_title":"Exploring the limits of transfer learning with a unified text-to-text transformer[J]","cited_arxiv_id":null,"evidence_quote":"The T5 text-to-text transformer used as the architecture for the fine-tuned intent extractor."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"MultiWOZ 2.1, one of the two datasets used for few-shot evaluation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"MultiWOZ 2.4, the corrected dataset used as the second benchmark."},{"cited_title":"Word-based dialog state tracking with recurrent neural networks[C]//Proceedings of the 15th annual meet- ing of the special interest group on discourse and dialogue (SIGDIAL)","cited_arxiv_id":null,"evidence_quote":"Joint Goal Accuracy, the evaluation metric for all main results."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Sentence-BERT, the retriever backbone fine-tuned for intent-driven example selection."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The strongest prior few-shot DST baseline that IDIC-DST claims to surpass."}],"review_version":1}