Pith. sign in

REVIEW 4 major objections 6 minor 40 references

Improving Dialogue State Tracking through Combinatorial Search for In-Context Examples

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read CombiSearch claims that scoring in-context examples by their combinatorial effect on dialogue-state accuracy, rather than by their individual similarity to the query, produces retrievers that beat state-of-the-art models with 5% of the…

desk verdict CombiSearch is a solid empirical contribution to retriever-based ICL for DST: the combinatorial scoring mechanism is real and isolated by ablations, but the M=3 scoring noise and missing variance estimates are the main soft spots. read the letter →

arxiv 2506.00622 v2 pith:JSOQQCKD submitted 2025-05-31 cs.CL cs.AIcs.IR

Haesung Pyun , Yoonah Park , Yohan Jo This is my paper
classification cs.CLcs.AIcs.IR
keywords dialoguestatetrackingin-contextlearningexampleretrievalcombinatorialsearchjointgoalaccuracydataefficiencyMultiWOZcontrastiveretrievertraining
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that the standard way of preparing training data for dialogue-state-tracking (DST) retrievers is the bottleneck: scoring each candidate example alone by its similarity to the query neglects how examples work together and ignores the query's own conversational structure. CombiSearch instead scores each example by its combinatorial track record—how often it appears in randomly sampled sets of examples that let an LLM predict the query's state change exactly. Retrievers trained on these scores beat all compared baselines on MultiWOZ 2.4 while using only 5% of the training data, and the same scored data lifts the oracle (no-retrieval-error) ceiling by 12 percentage points in Joint Goal Accuracy. The direct corollary is that a large part of DST performance is determined before retrieval happens, by the quality of the data used to train the retriever.

What carries the argument

The load-bearing object is the CombiScore, an accumulated per-example credit defined in Algorithm 1. For a query, a candidate pool is built by taking the union of BM25 and SBERT top-N retrievals and re-ranking by the product of the two normalized scores; then M times a random k-subset is drawn, the LLM is prompted with it, and a JGA of 1 adds one point to every example in that subset. The score thereby measures how often an example belongs to a winning combination, which is the mechanism that captures synergy and makes the scoring linear in the pool size rather than combinatorial. The resulting ranked pools are used to train a contrastive retriever (InfoNCE) whose positives are top-CombiScore examples and whose negatives are bottom-of-pool and out-of-pool examples.

What would settle it

Compute CombiScore for a fixed set of queries at M=3 and at a much larger budget (say M=50) and measure the rank correlation between the two score lists. If the correlation is weak, or if the top examples chosen at M=3 perform no better than random examples in the oracle setting, then the additive credit signal at the paper's operating point is mostly noise rather than a reliable measure of example utility. A cheaper check already visible in the paper: Table 14 shows oracle JGA still rising steeply from M=3 to M=10, so a reader can test whether a practical retriever trained on M=10 data beats one trained on M=3 data—a comparison the paper does not report.

Watch

Extended reading notes

Core claim

The central claim is that an example's usefulness for in-context learning in DST is a property of the teams it can join, not of its isolated similarity to the query. CombiSearch operationalizes this as CombiScore: for each query, it samples M random k-subsets from a diverse candidate pool, runs the LLM on the query with each subset, and increments the score of every example in every subset that yields a fully correct dialogue-state change. Because the credit signal is collected from successful combinations, the ranking reflects synergy among examples and is directly aligned with the evaluation metric, unlike re-ranking by dialogue-state overlap. The paper reports that this signal transfers: a retriever trained on CombiSearch-scored data from 5% of MultiWOZ outperforms all baselines trained on the full dataset, and the same scored data raises the oracle upper bound to 82.7% JGA with an 8B model versus 69.7% for the state-of-the-art data-preparation method.

Load-bearing premise

The method assumes that CombiScore accumulated from just three random combinations per example reliably ranks how much each example helps the LLM, even though the paper's own appendix shows that raising the evaluation count from three to ten moves the oracle score from 75.3 to 80.9 JGA, so the cheap ranking is not converged.

Editorial extensions

If this is right

  • Data efficiency: a retriever trained on CombiSearch-scored examples from 5% of MultiWOZ 2.4 reaches 59.8-64.2% JGA depending on the DST model, beating every baseline trained on 100% of the data, so DST systems can be built with a twentieth of the labeled dialogues.
  • Headroom: the oracle upper bound of 82.7% JGA (8B Llama-3) versus 69.7% for RefPyDST's data shows existing retrievers are not the only limiter—the training data they are fitted to is suboptimal, and improving that data is a direct route to higher practical performance.
  • Combinatorial signal: scoring examples in groups beats scoring them individually even at one-third the compute (82.7 vs 79.9 oracle JGA at M=3), and more evaluation budget widens the gap (85.3 at M=9), implying interactions among examples are a genuine, exploitable effect.
  • Diversity in the pool matters: replacing the hybrid BM25+SBERT pool with a random pool drops oracle JGA by 14 points, so the combinatorial scorer is only as good as the candidate pool it is applied to.
  • Transfer: the advantage persists on the SGD dataset, both when the retriever is trained in-domain and when a MultiWOZ-trained retriever is applied cross-dataset, so the scoring signal is not overfit to a single corpus.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The all-or-nothing JGA credit could be replaced by partial credit (e.g., per-slot accuracy), which might stabilize the low-M ranking and reduce the sensitivity documented in Table 14; testing this would require modifying Algorithm 1's credit rule.
  • Because the scoring LLM (Llama-3-8B) differs from the deployment LLMs (gpt-3.5-turbo, Llama-3-70B), the CombiScore is implicitly a transferable proxy; a testable extension is to measure how rankings change when the scoring model is matched to the deployment model or replaced by an ensemble.
  • The 12-point oracle gap suggests that retrievers themselves are near a ceiling once training data is fixed; a natural next step is to combine CombiSearch-ranked data with more expressive retriever training (e.g., iterative or RL-based retrieval), which the paper names as future work.
  • The random-subset sampler treats all combinations as equally informative; an active sampling strategy that seeks the most informative or diverse subsets could reduce the number of LLM calls needed to reach a converged ranking, a cost that currently scales linearly with M.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper introduces CombiSearch, a two-stage method for constructing retriever training data for in-context learning in dialogue state tracking. For each training query, CombiSearch first builds a diverse candidate pool by merging BM25 and SBERT retrievals and re-ranking with a hybrid score, then scores each candidate by the accumulated JGA of random k-subsets that contain it (termed CombiScore), using a small LLM as the DST model. The top-scoring examples are used as positives to fine-tune a contrastive retriever. Experiments on MultiWOZ 2.1/2.3/2.4 and SGD report improved JGA at 1%, 5%, and 100% data budgets, a claim of 20x data efficiency (5% data beating 100% baselines on MultiWOZ 2.4), and an oracle upper-bound gain of about 12% JGA over the previous SOTA when retrieval errors are removed.

Significance. The paper is a well-executed empirical study with a clearly specified method, extensive ablations, and a public code release. The cross-model transfer (scoring with Llama-3-8B, inference with gpt-3.5-turbo) is an important non-circularity check. The oracle upper-bound analysis and the accompanying error taxonomy (DST-EA) are useful contributions in their own right. If the M=3 scoring mechanism can be shown to be sufficiently reliable, or is replaced by a larger-M setting, the practical gains (5% data outperforming 100% baselines on MultiWOZ 2.4) would be notable for the ICL-for-DST community.

major comments (4)
  1. [§4 Common Settings, Appendix E.2 (Table 14)] The default evaluation budget M=3 yields highly noisy CombiScores. With N=100, k=10, and M=3 in Algorithm 1, each example is sampled at least once with probability 1−0.9^3 = 0.271, so roughly 73% of the candidate pool receives CombiScore 0 and the |P|=10 positive examples must come from the remaining sampled subset. Table 14 shows that increasing M from 3 to 10 raises oracle JGA at 5% data from 75.27 to 80.87, a 5.6-point swing, indicating that the M=3 ranking is far from converged. Since the headline 20x data-efficiency and 12% upper-bound claims are based on retrievers trained on M=3 scores, the paper needs to either use a larger M for the main results or provide evidence that the practical gains are robust to this noise, for example by comparing against a control where positive examples are selected at random from the same pool.
  2. [§4.4 (Table 7)] The comparison between Individual scoring and CombiSearch does not hold the LLM-call budget constant: Individual (100 candidates) uses 100 calls per query, CombiSearch (M=3) uses 30, and CombiSearch (M=9) uses 90. The stated conclusion that CombiSearch 'improves JGA by 2.8% over Individual while cutting compute to one-third' conflates the scoring method with the compute budget, because the M=9 condition is the one matched to the Individual budget. An equal-budget comparison (e.g., Individual with 30 candidates vs. CombiSearch with M=3, or Individual with 100 vs. CombiSearch with M=9) would more directly isolate the benefit of combinatorial scoring.
  3. [§3.2, §4.1.1 (Tables 1-2)] The data-efficiency claim lacks a control that isolates the CombiScore signal in retriever training. CombiSearch-trained retrievers are compared against RefPyDST and other baselines that use different positive-example definitions (state-overlap similarity plus diversity). Because most CombiScores are zero and ties are broken by dialogue-state overlap (§3.2), the retriever may be learning a similar state-overlap signal rather than the combinatorial one. An ablation that trains the same retriever on the same hybrid pool but with positives selected by random sampling or by hybrid score would clarify whether the gains come from the combinatorial scoring or from the pool construction and retriever architecture.
  4. [§4.2 (Table 5), Abstract] The abstract's '12% absolute improvement in the upper bound' refers to the comparison against RefPyDST, which differs from CombiSearch in multiple components (pool construction, prompt format, retriever training data). The isolated effect of combinatorial scoring on the oracle upper bound is the CombiSearch-versus-Hybrid comparison in Table 5: about 6.8 points for the 8B model and 12.0 points for the 70B model. The text should distinguish the total improvement over the previous SOTA from the contribution specifically attributable to combinatorial scoring, and the abstract should state which comparison the 12% figure refers to.
minor comments (6)
  1. [Algorithm 1, line 15] In Algorithm 1, the loop 'for e ∈ Pi' should read 'for e ∈ Ri' (the randomly sampled subset); as written, the loop iterates over the input prompt rather than the selected examples.
  2. [§4.2] The phrase 'no retrieval errors are assumed' is imprecise; the oracle setting assumes access to the query's ground-truth dialogue state, which is stronger than merely removing retrieval errors because it also supplies the gold state change for pool construction. Consider renaming this to 'gold-state oracle setting' or similar.
  3. [Table 1 note] The note that RefPyDST was 'reproduced under our own settings' should spell out the specific changes (e.g., beam search in place of PMI decoding) and report any difference from the originally published numbers, as this is the main ICL baseline.
  4. [Abstract] The phrase '20x gain in data efficiency' is based on comparing 5% of training data against 100% baselines; consider phrasing it as 'achieving the same or better JGA with 5% of the training data' to avoid implying a continuous scaling curve.
  5. [§4.3] The Se2 comparison in Table 6 uses N=30 for CombiSearch and N=50 for Se2, and 30 vs. 150 evaluations per query; clarify whether these numbers are directly comparable and why the smaller pool is an appropriate handicap.
  6. [§1] The sentence 'all these advances from CombiSearch's combinatorial nature come with a running time that is linear in the number of examples to score' is per-query linear; the total time scales with the number of query-example pairs, as reflected in Table 9. Consider making this distinction explicit.

Circularity Check

1 steps flagged · score 6.0 of 10

Oracle upper-bound claim is partly self-definitional: CombiScore is literally accumulated JGA, so the 12% oracle headroom measures the optimized objective; the practical retriever gains are independent and non-circular.

  1. self definitional [§3.1 Example Scoring (Algorithm 1 lines 12-18); §4.2 Upper Bound without Retrieval Errors]
    "A JGA score of 1 increases the score of every example in the sampled combination by 1, while a score of 0 leaves the scores unchanged. Repeating this sampling process M times yields an accumulated score for each example—termed CombiScore—reflecting its contribution to DST. ... we directly select the highest-scoring examples according to CombiScore with access to the query’s ground-truth dialogue state."

    CombiScore is, by Algorithm 1, the sum of JGA rewards assigned to examples during scoring. The oracle upper bound in §4.2 ranks the pool by CombiScore and then measures JGA with the same LLM and the same test query whose ground-truth state was used to compute the score. Selecting examples to maximize an accumulated-JGA score and then reporting JGA is therefore a near-tautology: the output metric is the input objective. The 12% absolute improvement over RefPyDST's oracle is expected by design, since RefPyDST ranks by dialogue-state overlap (indirect supervision) while CombiSearch ranks by the target metric itself.

full rationale

Most of the CombiSearch pipeline is non-circular. Retriever training data is constructed from training queries; CombiScore is computed with Llama-3-8B-Instruct; inference is evaluated on held-out MultiWOZ and SGD test sets with gpt-3.5-turbo and Llama models, including cross-dataset transfer. The 20x data-efficiency claim depends on a trained retriever selecting examples at inference, not on direct CombiScore selection, and it generalizes across models and datasets. There is no load-bearing self-citation chain and no imported uniqueness theorem. The one genuine circular step is the oracle upper-bound experiment (§4.2, Table 5): the paper reports the JGA achieved by selecting the examples ranked highest by CombiScore, but CombiScore is defined as the accumulated JGA of the same query-model pair used in that evaluation. The 12% headroom over RefPyDST is thus partly by construction—an oracle that optimizes JGA directly must beat an oracle that optimizes dialogue-state similarity. A separate robustness concern, not itself circularity, is that with M=3 evaluations each example appears in only ~27% of random subsets, leaving most of the pool with score 0, and Table 14 shows that increasing M from 3 to 10 changes oracle JGA by 5.6 points; this indicates the scoring signal is noisy, but it does not make the practical claims circular.

Assumptions & free parameters 6 free parameters · 5 assumptions · 1 invented entities

The central claim rests on a small set of tuned hyperparameters (N, k, M, P, B), a heuristic Monte Carlo credit assignment that is not mathematically proven, and the standard assumption that the datasets and JGA metric are trustworthy. No new physical entities are posited; CombiScore is a ledger entry whose utility is demonstrated only in this paper.

free parameters (6)
  • N (candidate pool size) = 100
    Size of the candidate pool per query; affects pool diversity and scoring cost; tuned manually on validation (Appendix B.2).
  • k (combination size) = 10
    Number of in-context examples per random subset during scoring; also used at inference; tuned on validation.
  • M (evaluations per example) = 3
    Number of random subset evaluations per query; controls noise of CombiScore and compute cost; chosen as cost-accuracy trade-off (Appendix E.2).
  • |P| (number of positive examples per query) = 10
    Top-scoring examples used as positives for InfoNCE loss; set manually (Appendix C).
  • B (number of hard negatives) = 16
    Hard negative count per positive in retriever training; tuned manually (Appendix B.2).
  • Score aggregation method = multiplication of z-scored BM25 and SBERT scores
    Chosen over weighted sum, RRF, and union top-k based on validation JGA (Appendix E.3).
assumptions (5)
  • domain assumption In-context learning with relevant examples improves DST performance over zero-shot
    Assumed throughout; supported by prior work (Hu et al. 2022; Liu et al. 2022), but the paper does not re-derive it.
  • domain assumption Joint Goal Accuracy as measured by the LLM on the query is a reliable reward signal
    CombiScore uses JGA=1 for each sampled subset; relies on exact-match scoring of the LLM's output.
  • ad hoc to paper Random subset sampling with additive credit approximates an example's marginal contribution
    No proof is given that the Monte Carlo estimate converges to the true combinatorial value; this is the core heuristic of CombiSearch (Algorithm 1).
  • ad hoc to paper BM25 and SBERT scores are comparable after z-score normalization and can be multiplied
    The hybrid score uses the product of z-normalized scores; validated only empirically on validation (Appendix E.3).
  • domain assumption MultiWOZ and SGD ground-truth labels are correct
    All JGA calculations depend on dataset labels; no label-quality analysis is provided.
invented entities (1)
  • CombiScore
    purpose: Aggregated JGA-based credit for each candidate in-context example, used to rank candidates and build retriever training data
    Defined in Algorithm 1; its predictive value is demonstrated only in the paper's own experiments; no external validation is provided.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Improving Dialogue State Tracking through Combinatorial Search for In-Context Examples." pith.science (2026). https://pith.science/paper/JSOQQCKD

@misc{pith2026250600622,
  author       = {Pith},
  title        = {Pith review of: Improving Dialogue State Tracking through Combinatorial Search for In-Context Examples},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/JSOQQCKD}},
  note         = {Machine review of arXiv:2506.00622}
}
read the original abstract

In dialogue state tracking (DST), in-context learning comprises a retriever that selects labeled dialogues as in-context examples and a DST model that uses these examples to infer the dialogue state of the query dialogue. Existing methods for constructing training data for retrievers suffer from three key limitations: (1) the synergistic effect of examples is not considered, (2) the linguistic characteristics of the query are not sufficiently factored in, and (3) scoring is not directly optimized for DST performance. Consequently, the retriever can fail to retrieve examples that would substantially improve DST performance. To address these issues, we present CombiSearch, a method that scores effective in-context examples based on their combinatorial impact on DST performance. Our evaluation on MultiWOZ shows that retrievers trained with CombiSearch surpass state-of-the-art models, achieving a 20x gain in data efficiency and generalizing well to the SGD dataset. Moreover, CombiSearch attains a 12% absolute improvement in the upper bound DST performance over traditional approaches when no retrieval errors are assumed. This significantly increases the headroom for practical DST performance while demonstrating that existing methods rely on suboptimal data for retriever training.

Figures

Figures reproduced from arXiv: 2506.00622 by the authors.

Figure 1
Figure 1. (A) Conventional data preparation method. [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Overall structure of CombiSearch. identify an optimal set of examples that collectively enhance DST performance. However, exhaustively evaluating all possible combinations is impractical due to the combina￾torial explosion. Instead, we conceptualize each example as a key “team player” if it consistently contributes to the success of the “teams” it belongs to (i.e., accurate DST) across different sampled combinations… view at source ↗
Figure 3
Figure 3. Error tree to divide error type by slot and value [PITH_FULL_IMAGE:figures/full_fig_p015_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Statistics of error types made by different models in § [PITH_FULL_IMAGE:figures/full_fig_p015_4.png]
Figure 5
Figure 5. Figure 5: Changes in JGA according to the number of shots. From left to right: BM25, SBERT, and BM25&SBERT. [PITH_FULL_IMAGE:figures/full_fig_p018_5.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

40 extracted references · 11 canonical work pages

  1. [1]

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gr...

    2020

  2. [2]

    Ting-Yun Chang and Robin Jia. 2023. https://doi.org/10.18653/v1/2023.acl-long.452 Data curation alone can stabilize in-context learning . In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8123--8144, Toronto, Canada. Association for Computational Linguistics

    doi:10.18653/v1/2023.acl-long.452 2023

  3. [3]

    Derek Chen, Kun Qian, and Zhou Yu. 2023. https://doi.org/10.18653/v1/2023.findings-eacl.115 Stabilized in-context learning with pre-trained language models for few shot dialogue state tracking . In Findings of the Association for Computational Linguistics: EACL 2023, pages 1551--1564, Dubrovnik, Croatia. Association for Computational Linguistics

    work page doi:10.18653/v1/2023.findings-eacl.115 2023

  4. [4]

    Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradb...

    work page 2023

  5. [5]

    Cormack, Charles L A Clarke, and Stefan Buettcher

    Gordon V. Cormack, Charles L A Clarke, and Stefan Buettcher. 2009. https://doi.org/10.1145/1571941.1572114 Reciprocal rank fusion outperforms condorcet and individual rank learning methods . In Proceedings of the 32nd International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '09, page 758–759, New York, NY, USA. Associ...

    arXiv 2009

  6. [6]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783

    arXiv 2024

  7. [7]

    Yujie Feng, Zexin Lu, Bo Liu, Liming Zhan, and Xiao-Ming Wu. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.48 Towards LLM -driven dialogue state tracking . In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 739--755, Singapore. Association for Computational Linguistics

    doi:10.18653/v1/2023.emnlp-main.48 2023

  8. [8]

    Raghav Gupta, Harrison Lee, Jeffrey Zhao, Yuan Cao, Abhinav Rastogi, and Yonghui Wu. 2022. https://doi.org/10.18653/v1/2022.naacl-main.336 Show, don`t tell: Demonstrations outperform descriptions for schema-guided task-oriented dialogue . In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: ...

    doi:10.18653/v1/2022.naacl-main.336 2022

Show all 40 references
  1. [9]

    Shivanshu Gupta, Matt Gardner, and Sameer Singh. 2023. https://doi.org/10.18653/v1/2023.findings-emnlp.930 Coverage-based example selection for in-context learning . In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 13924--13950, Singapore. Associ...

    2023 doi

  2. [10]

    Ting Han, Ximing Liu, Ryuichi Takanobu, Yixin Lian, Chongxuan Huang, Dazhen Wan, Wei Peng, and Minlie Huang. 2021. https://arxiv.org/abs/2010.05594 Multiwoz 2.3: A multi-domain task-oriented dialogue dataset enhanced with annotation corrections and co-reference annotation . Pr...

    2021 arXiv

  3. [11]

    Jia He, Mukund Rungta, David Koleczek, Arshdeep Sekhon, Franklin X Wang, and Sadid Hasan. 2024. https://arxiv.org/abs/2411.10541 Does prompt formatting have any impact on llm performance? Preprint, arXiv:2411.10541

    2024 arXiv

  4. [12]

    Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katherine Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon ...

    2022

  5. [13]

    Ari Holtzman, Peter West, Vered Shwartz, Yejin Choi, and Luke Zettlemoyer. 2021. https://doi.org/10.18653/v1/2021.emnlp-main.564 Surface form competition: Why the highest probability answer isn`t always right . In Proceedings of the 2021 Conference on Empirical Methods in Natu...

    2021 doi

  6. [14]

    Ehsan Hosseini-Asl, Bryan McCann, Chien-Sheng Wu, Semih Yavuz, and Richard Socher. 2020. A simple language model for task-oriented dialogue. In Proceedings of the 34th International Conference on Neural Information Processing Systems, NIPS '20, Red Hook, NY, USA. Curran Associates Inc

    2020

  7. [15]

    Smith, and Mari Ostendorf

    Yushi Hu, Chia-Hsuan Lee, Tianbao Xie, Tao Yu, Noah A. Smith, and Mari Ostendorf. 2022. https://doi.org/10.18653/v1/2022.findings-emnlp.193 In-context learning for few-shot dialogue state tracking . In Findings of the Association for Computational Linguistics: EMNLP 2022, page...

    2022 doi

  8. [16]

    Dan Iter, Reid Pryzant, Ruochen Xu, Shuohang Wang, Yang Liu, Yichong Xu, and Chenguang Zhu. 2023. https://doi.org/10.18653/v1/2023.findings-emnlp.81 In-context demonstration selection with cross entropy difference . In Findings of the Association for Computational Linguistics:...

    2023 doi

  9. [17]

    Brendan King and Jeffrey Flanigan. 2023. https://doi.org/10.18653/v1/2023.findings-acl.344 Diverse retrieval-augmented in-context learning for dialogue state tracking . In Findings of the Association for Computational Linguistics: ACL 2023, pages 5570--5585, Toronto, Canada. A...

    2023 doi

  10. [18]

    Aobo Kong, Shiwan Zhao, Hao Chen, Qicheng Li, Yong Qin, Ruiqi Sun, Xin Zhou, Enzhi Wang, and Xiaohang Dong. 2024. https://doi.org/10.18653/v1/2024.naacl-long.228 Better zero-shot reasoning with role-play prompting . In Proceedings of the 2024 Conference of the North American C...

    2024 doi

  11. [19]

    Atharva Kulkarni, Bo-Hsiang Tseng, Joel Ruben Antony Moniz, Dhivya Piraviperumal, Hong Yu, and Shruti Bhargava. 2024. https://aclanthology.org/2024.eacl-long.120 S ynth DST : Synthetic data is all you need for few-shot dialog state tracking . In Proceedings of the 18th Confere...

    2024

  12. [20]

    Chia-Hsuan Lee, Hao Cheng, and Mari Ostendorf. 2021. https://doi.org/10.18653/v1/2021.emnlp-main.404 Dialogue state tracking with a language model using schema-driven prompting . In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 4...

    2021 doi

  13. [21]

    Itay Levy, Ben Bogin, and Jonathan Berant. 2023. https://doi.org/10.18653/v1/2023.acl-long.78 Diverse demonstrations improve in-context compositional generalization . In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pap...

    2023 doi

  14. [22]

    Haoyu Liu, Jianfeng Liu, Shaohan Huang, Yuefeng Zhan, Hao Sun, Weiwei Deng, Furu Wei, and Qi Zhang. 2024 a . https://doi.org/10.18653/v1/2024.findings-acl.312 se^2 : Sequential example selection for in-context learning . In Findings of the Association for Computational Linguis...

    2024 doi

  15. [23]

    Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, and Weizhu Chen. 2022. https://doi.org/10.18653/v1/2022.deelio-1.10 What makes good in-context examples for GPT -3? In Proceedings of Deep Learning Inside Out (DeeLIO 2022): The 3rd Workshop on Knowledge Extr...

    2022 doi

  16. [24]

    Jinxin Liu, Li He, Yachen Kang, Zifeng Zhuang, Donglin Wang, and Huazhe Xu. 2023. Ceil: generalized contextual imitation learning. In Proceedings of the 37th International Conference on Neural Information Processing Systems, NIPS '23, Red Hook, NY, USA. Curran Associates Inc

    2023

  17. [25]

    we need structured output

    Michael Xieyang Liu, Frederick Liu, Alexander J. Fiannaca, Terry Koo, Lucas Dixon, Michael Terry, and Carrie J. Cai. 2024 b . https://doi.org/10.1145/3613905.3650756 "we need structured output": Towards user-centered constraints on large language model output . In Extended Abs...

    2024

  18. [26]

    Pan Lu, Liang Qiu, Kai-Wei Chang, Ying Nian Wu, Song-Chun Zhu, Tanmay Rajpurohit, Peter Clark, and Ashwin Kalyan. 2023. https://openreview.net/forum?id=DHyHRBwJUTN Dynamic prompt learning via policy gradient for semi-structured mathematical reasoning . In ICLR

    2023

  19. [27]

    Tai Nguyen and Eric Wong. 2023. https://arxiv.org/abs/2302.11042 In-context example selection with influences . Preprint, arXiv:2302.11042

    2023 arXiv

  20. [28]

    Keqin Peng, Liang Ding, Yancheng Yuan, Xuebo Liu, Min Zhang, Yuanxin Ouyang, and Dacheng Tao. 2024. https://doi.org/10.18653/v1/2024.acl-long.492 Revisiting demonstration selection strategies in in-context learning . In Proceedings of the 62nd Annual Meeting of the Association...

    2024 doi

  21. [29]

    Nils Reimers and Iryna Gurevych. 2019. https://doi.org/10.18653/v1/D19-1410 Sentence- BERT : Sentence embeddings using S iamese BERT -networks . In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference...

    2019 doi

  22. [30]

    Stephen Robertson and Hugo Zaragoza. 2009. https://doi.org/10.1561/1500000019 The probabilistic relevance framework: Bm25 and beyond . Found. Trends Inf. Retr., 3(4):333–389

    2009 doi

  23. [31]

    Ohad Rubin, Jonathan Herzig, and Jonathan Berant. 2022. https://doi.org/10.18653/v1/2022.naacl-main.191 Learning to retrieve prompts for in-context learning . In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: ...

    2022 doi

  24. [32]

    Jamin Shin, Hangyeol Yu, Hyeongdon Moon, Andrea Madotto, and Juneyoung Park. 2022. https://doi.org/10.18653/v1/2022.findings-acl.302 Dialogue summaries as dialogue states ( DS 2), template-guided summarization for few-shot dialogue state tracking . In Findings of the Associati...

    2022 doi

  25. [33]

    Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tie-Yan Liu. 2020. https://arxiv.org/abs/2004.09297 Mpnet: Masked and permuted pre-training for language understanding . Preprint, arXiv:2004.09297

    2020 arXiv

  26. [34]

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, W...

    2023 arXiv

  27. [35]

    Aaron van den Oord, Yazhe Li, and Oriol Vinyals. 2019. https://arxiv.org/abs/1807.03748 Representation learning with contrastive predictive coding . Preprint, arXiv:1807.03748

    2019 arXiv

  28. [36]

    Praveen Venkateswaran, Evelyn Duesterwald, and Vatche Isahagian. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.310 D i STRICT : Dialogue state tracking with retriever driven in-context tuning . In Proceedings of the 2023 Conference on Empirical Methods in Natural Language ...

    2023 doi

  29. [37]

    Chien-Sheng Wu, Andrea Madotto, Ehsan Hosseini-Asl, Caiming Xiong, Richard Socher, and Pascale Fung. 2019. https://doi.org/10.18653/v1/P19-1078 Transferable multi-domain state generator for task-oriented dialogue systems . In Proceedings of the 57th Annual Meeting of the Assoc...

    2019 doi

  30. [38]

    Jiacheng Ye, Zhiyong Wu, Jiangtao Feng, Tao Yu, and Lingpeng Kong. 2023. Compositional exemplars for in-context learning. In Proceedings of the 40th International Conference on Machine Learning, ICML'23. JMLR.org

    2023

  31. [39]

    Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. 2022. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068

    2022 arXiv

  32. [40]

    Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, et al. 2023. A survey of large language models. arXiv preprint arXiv:2303.18223, 1(2)

    2023 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.