REVIEW 5 major objections 4 minor 28 references
Intent-driven In-context Learning for Few-shot Dialogue State Tracking
T0 review · 5 major / 4 minor · reviewed 2026-08-11 · deepseek-v4-flash
Pith's one-line read Intent-driven in-context learning lifts few-shot dialogue state tracking to state of the art.
desk verdict The intent-driven retrieval idea is sensible, but the central SOTA claim is unverifiable because the same configuration scores 52.66 in Table II and 43.39 in Table IV, and the method leaks its own predicted intent into the prompt as a reference output. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The mechanism is a two-module pipeline built around the user's intent. The Intent-driven Dialogue Information Augmentation module fine-tunes a T5 model to extract the current intent and concatenates it with the dialogue context, giving the LLM explicit access to what the user actually wants. The Intent-driven Examples Retrieval module masks dialogue history, current state, and previous state from the augmented information, rewrites the user input in terms of the intent, and uses a fine-tuned Sentence-BERT retriever to find the top-k similar examples by an average F1 score over slot sets and slot-value sets. The final component casts dialogue state updating as text-to-SQL generation: a fixed SQL command initializes each domain as a table, and the LLM's generated SQL query is parsed into state changes. The intent is the load-bearing representation: it cleans the retrieval query and supplies the reference signal that lets the generator handle implicit language.
What would settle it
Inspect the data split used for the 1% sample; if any evaluation dialogue appears in the training sample or the retrieval pool, re-run the experiments with a strictly training-only sample and pool. A substantial drop in the reported 43.11 and 52.66 joint goal accuracy under that clean split would falsify the claim that intent-driven in-context learning alone produces the state-of-the-art results.
Extended reading notes
Core claim
The central claim is that the bottleneck in few-shot DST is the information put into the prompt, not the generative model. The authors find that examples retrieved using only the user's current intent give better guidance than examples retrieved using dialogue history or previous state, because history contains too much irrelevant content. They also find that appending the extracted intent to the dialogue context helps the LLM resolve implicit user inputs. Based on these findings, the proposed system augments the dialogue representation with the intent, masks history and prior state tokens before retrieval, rewrites the user input into an explicit form driven by the intent, and fine-tunes a sentence-embedding retriever with a slot/slot-value similarity score. The final prompt is formatted as text-to-SQL, which makes the state update easy to parse. The reported outcome is state-of-the-art joint goal accuracy on both MultiWOZ 2.1 and MultiWOZ 2.4 at 1% few-shot, with gains over the prior best system and parity or better relative to a 175-billion-parameter baseline.
Load-bearing premise
The load-bearing assumption is that the 1% of MultiWOZ used to train the intent extractor and the retriever and to form the retrieval pool comes only from the training split; if evaluation dialogues are included in that 1%, the reported accuracy gains could come from retrieval leakage rather than from the method.
Editorial extensions
If this is right
- At 1% few-shot, IDIC-DST reports joint goal accuracy of 43.11 on MultiWOZ 2.1 and 52.66 on MultiWOZ 2.4, outperforming the previous best few-shot DST models in the comparison.
- The method achieves this with a 7-billion-parameter generator, matching or beating a 175-billion-parameter in-context baseline, which suggests parameter scale is not the decisive factor in few-shot DST.
- Adding the intent-driven dialogue information augmentation alone accounts for a large share of the gains over the base in-context learner, and adding the intent-driven retrieval on top produces the full improvement.
- Reformulating state updates as SQL generation gives a structured, parseable representation that can be applied to update prior dialogue state.
- In an end-to-end task-oriented dialogue system, the improved state tracker also raises booking rate, completion rate, and F1 while reducing average turns.
Reading between the lines
- Because both auxiliary modules are trained on the same 1% sample used for the few-shot evaluation, the method's 'few-shot' budget is larger than the name suggests; a comparison that charges all methods for every trained component could change the ranking.
- The finding that dialogue history hurts retrieval suggests a general pre-retrieval denoising step: strip historical context and surface the underlying intent before embedding, which could transfer to other retrieval-augmented generation tasks.
- The text-to-SQL formulation is a natural fit for slot-value updates and could be reused for any structured schema-tracking problem, such as form filling or API argument tracking.
- If the 1% sample includes evaluation dialogues, the reported numbers would be inflated; the paper's split choice should be verified to settle the claim.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes IDIC-DST, a few-shot dialogue state tracking (DST) method built on CodeLlama-7B. A fine-tuned T5-small model first extracts the user's intent and augments the dialogue information; an SBERT retriever then retrieves top-k in-context examples from a masked and rewritten version of the augmented dialogue; the LLM is prompted with these examples plus the augmented information to generate a SQL query that updates the dialogue state. The authors report state-of-the-art joint goal accuracy (JGA) on MultiWOZ 2.1 and MultiWOZ 2.4 under a 1% few-shot setting (43.11 and 52.66, respectively) and include ablation, task-oriented dialogue system, and case-study analyses.
Significance. If the reported results are correct, IDIC-DST would be a useful contribution: it addresses two recognized difficulties in few-shot DST, namely implicit user input and noisy dialogue history, and it does so with a relatively small generation model (7B parameters). The text-to-SQL formulation is an interesting alternative to the code-generation framing of prior in-context-learning DST work, and the case study (Table V) illustrates the intended behavior concretely. However, the manuscript as written does not provide reliable evidence for its central claim: there is an unexplained internal contradiction in the main results, the 1% data split is underspecified with a serious leakage risk, the design includes the supervised intent prediction as a reference output in the LLM prompt, and no variance or seed information is reported. These issues block verification of the state-of-the-art claim.
major comments (5)
- [Table II vs. Table IV] Table II reports IDIC-DST at 52.66 JGA on 1% MultiWOZ 2.4, while Table IV reports 43.39 JGA for the row labeled 'w/ DCA,IER', which Section III-D explicitly describes as the full method after adding both proposed modules to IC-DST. On the same 1% setting these two entries should describe the same system and should match; the 9.27-point difference is unexplained. The paper does not state whether the two tables use the same 1% sample, the same retriever, the same value of k, the same prompt template, or the same evaluation split. This internal contradiction makes the central state-of-the-art claim unverifiable as written.
- [Section III-B] The text states 'we extracted 1% of the MultiWOZ dataset' for training the T5 NLU model, the SBERT retriever, and for use as the retrieval sample pool, but it does not specify that this 1% is drawn exclusively from the training split and excludes evaluation dialogues. Because the same 1% is used for supervised training, retrieval fine-tuning, and the retrieval pool, any overlap with test dialogues would directly inflate JGA through memorization and retrieval leakage. The authors must specify the exact split, verify no test-set contamination, and ideally report results across multiple random 1% samples.
- [Section II-C and Table V] The augmented dialogue information D'_t includes the T5-extracted intent It, and Eq. (3) shows that It remains unmasked in D''_t; Section II-C then feeds D''_t into the LLM prompt. The case study in Table V makes the mechanism explicit: 'IDIC-DST referance output: (attraction-area:south)' is provided to the LLM, and the final output is exactly that value. The LLM's final dialogue state may therefore be largely a copy of the supervised T5 prediction, so the observed gains in Table IV cannot be attributed to in-context learning or retrieval alone. The authors should report an ablation that removes It from the LLM prompt while keeping it for retrieval, and should also report the JGA of the T5 intent extractor alone as a reference point.
- [Tables II and IV] All JGA numbers appear to come from a single run with no variance or number of seeds reported. In a few-shot setting with only 1% of the data, results are highly sensitive to the random split, model initialization, and retrieval order; for example, Table II shows a 0.02-point difference between IDIC-DST (43.11) and IC-DST Codex (43.13) on MultiWOZ 2.1, which cannot be interpreted without error bars. The authors should report the mean and standard deviation over at least 3–5 seeds and indicate whether the differences are statistically significant.
- [Sections II-B and III-B] The description of the retrieval module and the experimental setup omits several details needed for reproducibility: the value of k, the exact rewriting procedure for the user's input, the masking implementation, the full prompt template for CodeLlama-7B, and decoding parameters (temperature, max length, sampling strategy). Without these details, the reported experiments cannot be reproduced or independently checked.
minor comments (4)
- [Table V] The table contains typos: 'referance' should be 'reference', and 'WRONG MASSAGES' should be 'WRONG MESSAGES'.
- [Section III-D and Table IV] The abbreviation for the Intent-driven Dialogue Information Augmentation module is inconsistent: the text uses 'IDA' while Table IV uses 'DCA' in the row labels; the table should use the same acronym as the text.
- [Table II] SM2-11B is listed in both the '1–10B' and the '10–100B' parameter-size groups, with different JGA values in each row; since 11B belongs in the latter group, the duplicated row is confusing and should be removed or corrected.
- [Section III-C] The sentence describing the TOD system says 'the natural language understanding module employs a template-based generation method'; this appears to be a typo for 'natural language generation module', and the roles of the policy and NLG modules should be clarified.
Circularity Check
IDIC-DST's central few-shot result is partially circular: the intent extractor's output It is inserted into the LLM prompt as a 'reference output', and the final dialogue state copies it; retrieval also uses It, which is a subset of the target state change, as its query.
-
fitted input called prediction
[Section II-A (Eq. 2), Section III-E, Table V]
"In contrast, IDIC-DST not only retrieves It and rewrites the user's input based on it, thereby retrieving more suitable in-context examples, but also provide It as a reference output to the LLM, enabling it to perform well even when the user's input is implicit. [Table V:] IDIC-DST referance output : (attraction-area:south) ... IDIC-DST final output : (attraction-area:south)"
The T5 model is fine-tuned on the few-shot data to output It, a set of slot-value pairs that is exactly the current-turn dialogue-state change (e.g., attraction-area:south). This It is then placed into the LLM prompt as a 'reference output', and Table V shows the final LLM output is literally the same value. Thus, for the slots covered by It, the reported joint goal accuracy is not an independent LLM prediction; it is a copy of the supervised T5 extractor's output. The LLM is not being tested on those slots, and the pipeline's success is forced by the fitted T5 prediction rather than by in-context learning.
-
self definitional
[Section II-B, Eq. (3) and Eq. (6)]
"we mask dialogue history, Bt and Bt−1 before retrieval to obtain the rewritten information D''t: D''t = Mask(D't) = {turn, domain, [MASK], [MASK], It, [MASK]}. ... The retriever is then fine-tuned using contrastive loss to ensure high similarity scores between similar sets of state changes."
The retrieval query D''t has It as its only unmasked semantic content, and It is a slot-value representation of the current turn's state change. The retriever is explicitly trained to score similarity between sets of state changes (Eq. 6), so the examples selected are those whose slot-value labels are closest to the current turn's target label. Retrieval therefore uses the answer to find demonstrations that contain the answer, and those demonstrations are then fed to the LLM to generate the same state change. The retrieval step is self-referential: the target state change defines the query that selects the examples used to predict it.
full rationale
IDIC-DST's reported few-shot JGA is not fully self-contained. The intent extractor's output It is both part of the input augmentation and a 'reference output' inserted into the LLM prompt; Table V shows the final state is identical to that reference. Because It is a slot-value subset of the target dialogue state, the LLM's output on those slots reduces to the supervised T5's prediction, and the retrieval in Section II-B selects demonstrations by similarity of state changes, i.e., by the label. This is a construction-level reduction rather than a self-citation issue. There is no load-bearing self-citation: reference [1] is a same-author survey but is not used to justify the method. The unexplained gap between Table II (52.66) and Table IV (43.39) for the same full method is a serious verifiability problem but is not itself a circularity step. Overall, the central claim is partially circular, so the score is 6.
Assumptions & free parameters
free parameters (1)
- k (number of retrieved examples)
assumptions (3)
- domain assumption The user's intent It at turn t is adequately represented by the set of slot-value pairs that change in that turn, derived from MultiWOZ state annotations.
- domain assumption The fine-tuned T5-small and SBERT models, trained on 1% of MultiWOZ, generalize to the evaluation split without overlapping it.
- standard math The LLM (CodeLlama 7B) follows the text-to-SQL instruction and produces parseable SQL updates.
Cite this review
Pith. "Pith review of Intent-driven In-context Learning for Few-shot Dialogue State Tracking." pith.science (2026). https://pith.science/paper/2JHJLFZI
@misc{pith2026241203270,
author = {Pith},
title = {Pith review of: Intent-driven In-context Learning for Few-shot Dialogue State Tracking},
year = {2026},
howpublished = {\url{https://pith.science/paper/2JHJLFZI}},
note = {Machine review of arXiv:2412.03270}
}
read the original abstract
Dialogue state tracking (DST) plays an essential role in task-oriented dialogue systems. However, user's input may contain implicit information, posing significant challenges for DST tasks. Additionally, DST data includes complex information, which not only contains a large amount of noise unrelated to the current turn, but also makes constructing DST datasets expensive. To address these challenges, we introduce Intent-driven In-context Learning for Few-shot DST (IDIC-DST). By extracting user's intent, we propose an Intent-driven Dialogue Information Augmentation module to augment the dialogue information, which can track dialogue states more effectively. Moreover, we mask noisy information from DST data and rewrite user's input in the Intent-driven Examples Retrieval module, where we retrieve similar examples. We then utilize a pre-trained large language model to update the dialogue state using the augmented dialogue information and examples. Experimental results demonstrate that IDIC-DST achieves state-of-the-art performance in few-shot settings on MultiWOZ 2.1 and MultiWOZ 2.4 datasets.
Figures
Reference graph
Works this paper leans on
-
[1]
A Survey on Recent Ad- vances in LLM-Based Multi-turn Dialogue Systems[J]
Yi Z, Ouyang J, Liu Y , et al. A Survey on Recent Ad- vances in LLM-Based Multi-turn Dialogue Systems[J]. arXiv preprint arXiv:2402.18013, 2024
arXiv 2024
-
[2]
Jinjie Ni, Tom Young, Vlad Pandelea, Fuzhao Xue, and Erik Cambria
-
[6]
Puhai Yang, Heyan Huang, Shumin Shi, and Xian-Ling Mao. 2024. STN4DST: A Scalable Dialogue State Tracking Based on Slot Tagging Navigation. IEEE/ACM Trans. Audio, Speech and Lang. Proc. 32 (2024), 2494–2507. https://doi.org/10.1109/TASLP.2024.3393733
arXiv 2024
-
[7]
Yifan Wang, Jing Zhao, Junwei Bao, Chaoqun Duan, Youzheng Wu, and Xiaodong He. 2022. LUNA: Learning Slot-Turn Alignment for Dialogue State Tracking. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3319–3328, Seattle, United States. Association for Comp...
work page 2022
-
[8]
Brown T B, Mann B, Ryder N, et al. Language models are few-shot learners[C]//Proceedings of the 34th International Conference on Neural Information Processing Systems. 2020: 1877-1901
work page 2020
-
[9]
Atharva Kulkarni, Bo-Hsiang Tseng, Joel Ruben Antony Moniz, Dhivya Piraviperumal, Hong Yu, and Shruti Bhargava. 2024. SynthDST: Syn- thetic Data is All You Need for Few-Shot Dialog State Tracking. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (V olume 1: Long Papers), pages 1988–2001, St. Ju...
work page 2024
-
[10]
Yuting Yang, Wenqiang Lei, Pei Huang, Juan Cao, Jintao Li, and Tat- Seng Chua. 2023. A Dual Prompt Learning Framework for Few-Shot Dialogue State Tracking. In Proceedings of the ACM Web Conference 2023 (WWW ’23). Association for Computing Machinery, New York, NY , USA, 1468–1477. https://doi.org/10.1145/3543507.3583238
arXiv 2023
-
[11]
R. Su, J. Yang, T. -W. Wu and B. -H. Juang, ”Choice Fusion As Knowledge For Zero-Shot Dialogue State Tracking,” ICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Rhodes Island, Greece, 2023, pp. 1-5, doi: 10.1109/ICASSP49357.2023.10096669. keywords: Train- ing;Costs;Fuses;Signal processing;Data models;De...
arXiv 2023
Show all 28 references
-
[12]
Seanie Lee, Jianpeng Cheng, Joris Driesen, Alexandru Coca, and Anders Johannsen. 2024. Effective and Efficient Conversation Retrieval for Di- alogue State Tracking with Implicit Text Summaries. In Proceedings of the 2024 Conference of the North American Chapter of the Associat...
2024
-
[13]
A survey of hallucination in large foundation models[J]
Rawte V , Sheth A, Das A. A survey of hallucination in large foundation models[J]. arXiv preprint arXiv:2309.05922, 2023
2023 arXiv
-
[14]
Smith, and Mari Ostendorf
Yushi Hu, Chia-Hsuan Lee, Tianbao Xie, Tao Yu, Noah A. Smith, and Mari Ostendorf. 2022. In-Context Learning for Few-Shot Dialogue State Tracking. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 2627–2643, Abu Dhabi, United Arab Emirates. Associa...
2022
-
[15]
Brendan King and Jeffrey Flanigan. 2023. Diverse Retrieval-Augmented In-Context Learning for Dialogue State Tracking. In Findings of the As- sociation for Computational Linguistics: ACL 2023, pages 5570–5585, Toronto, Canada. Association for Computational Linguistics
2023
-
[16]
Exploring the limits of transfer learning with a unified text-to-text transformer[J]
Raffel C, Shazeer N, Roberts A, et al. Exploring the limits of transfer learning with a unified text-to-text transformer[J]. Journal of machine learning research, 2020, 21(140): 1-67
2020
-
[17]
Budzianowski P, Wen T H, Tseng B H, et al. MultiWOZ-A Large-Scale Multi-Domain Wizard-of-Oz Dataset for Task-Oriented Dialogue Mod- elling[C]//Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. 2018: 5016-5026
2018
-
[18]
Ye F, Manotumruksa J, Yilmaz E. MultiWOZ 2.4: A Multi-Domain Task-Oriented Dialogue Dataset with Essential Annotation Corrections to Improve State Tracking Evaluation[C]//Proceedings of the 23rd An- nual Meeting of the Special Interest Group on Discourse and Dialogue. 2022: 351-360
2022
-
[19]
Word-based dialog state tracking with recurrent neural networks[C]//Proceedings of the 15th annual meet- ing of the special interest group on discourse and dialogue (SIGDIAL)
Henderson M, Thomson B, Young S. Word-based dialog state tracking with recurrent neural networks[C]//Proceedings of the 15th annual meet- ing of the special interest group on discourse and dialogue (SIGDIAL). 2014: 292-299
2014
-
[20]
Nils Reimers and Iryna Gurevych. 2019. Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJC...
2019
-
[21]
Code llama: Open foundation models for code[J]
Roziere B, Gehring J, Gloeckle F, et al. Code llama: Open foundation models for code[J]. arXiv preprint arXiv:2308.12950, 2023
2023 arXiv
-
[22]
Chien-Sheng Wu, Andrea Madotto, Ehsan Hosseini-Asl, Caiming Xiong, Richard Socher, and Pascale Fung. 2019. Transferable Multi-Domain State Generator for Task-Oriented Dialogue Systems. In Proceedings of the 57th Annual Meeting of the Association for Computational Lin- guistics...
2019
-
[23]
Dialogue State Tracking with a Language Model using Schema-Driven Prompting[C]//Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing
Lee C H, Cheng H, Ostendorf M. Dialogue State Tracking with a Language Model using Schema-Driven Prompting[C]//Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. 2021: 4937-4949
2021
-
[24]
Shin J, Yu H, Moon H, et al. Dialogue Summaries as Dialogue States (DS2), Template-Guided Summarization for Few-shot Dialogue State Tracking[C]//Findings of the Association for Computational Linguistics: ACL 2022. 2022: 3824-3846
2022
-
[25]
S. Feng, H. Su, H. Xie, D. Wu, H. Huang and W. Silamu, ”Fact-Aware Summarization with Contrastive Learning for Few- Shot Dialogue State Tracking,” ICASSP 2024 - 2024 IEEE In- ternational Conference on Acoustics, Speech and Signal Process- ing (ICASSP), Seoul, Korea, Republic o...
2024
-
[26]
Derek Chen, Kun Qian, and Zhou Yu. 2023. Stabilized In-Context Learning with Pre-trained Language Models for Few Shot Dialogue State Tracking. In Findings of the Association for Computational Linguistics: EACL 2023, pages 1551–1564, Dubrovnik, Croatia. Association for Computat...
2023
-
[27]
Towards LLM-driven Dialogue State Tracking[C]//Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing
Feng Y , Lu Z, Liu B, et al. Towards LLM-driven Dialogue State Tracking[C]//Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. 2023: 739-755
2023
-
[28]
Building Multi-domain Dialog State Trackers from Single-domain Dialogs[C]//Proceedings of the 2023 Con- ference on Empirical Methods in Natural Language Processing
Zhu Q, Zhang Z, Zhu X, et al. Building Multi-domain Dialog State Trackers from Single-domain Dialogs[C]//Proceedings of the 2023 Con- ference on Empirical Methods in Natural Language Processing. 2023: 15323-15335
2023
-
[29]
Christian Geishauser, Carel van Niekerk, Hsien-chin Lin, Nurul Lubis, Michael Heck, Shutong Feng, and Milica Ga ˇsi´c. 2022. Dynamic Dia- logue Policy for Continual Reinforcement Learning. In Proceedings of the 29th International Conference on Computational Linguistics, pages ...
2022
-
[30]
TripPy: A Triple Copy Strategy for Value Independent Neural Dialog State Tracking[C]//Proceedings of the 21th Annual Meeting of the Special Interest Group on Discourse and Dialogue
Heck M, van Niekerk C, Lubis N, et al. TripPy: A Triple Copy Strategy for Value Independent Neural Dialog State Tracking[C]//Proceedings of the 21th Annual Meeting of the Special Interest Group on Discourse and Dialogue. 2020: 35-44
2020
-
[2022]
Recent advances in deep learning based dialogue systems: a systematic survey. Artif. Intell. Rev. 56, 4 (Apr 2023), 3055–3155. https://doi.org/10.1007/s10462-022-10248-8
2023 doi
Reviewed August 11, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.