Pith. sign in

REVIEW 3 major objections 6 minor 43 references

Instruction-tuned 3B open-weight models can beat larger 7B base models on zero-shot intent classification, while SNIPS no longer ranks them.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Across eight zero-shot intent datasets, instruction-tuned ~3B open-weight models can match or beat larger base models, top systems are statistically tied on MASSIVE, and SNIPS is saturated.

T0 review reviewed 2026-07-31 challenge →

load-bearing objection Useful practitioner bake-off of 41 sub-9B open-weight models on zero-shot IC; the IT-over-scale headline is real under their free-text protocol but partly confounded with format compliance. the 3 major comments →

arxiv 2607.27421 v1 pith:Y2NTYWIE submitted 2026-07-29 cs.CL cs.AIcs.LG

Selecting Open-Weight Language Models for Zero-Shot Intent Classification: A Systematic Evaluation of 41 Models

Parishruthi Ganesh , Gerry Dozier , Cheryl Seals This is my paper
classification cs.CL cs.AIcs.LG
keywords intent classificationzero-shot learningopen-weight language modelsinstruction tuningconfidence calibrationbenchmark saturationdeployment efficiencyrobustness
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Practitioners who ship task-oriented dialogue systems need to pick an open-weight language model that fits real compute, latency, and noise constraints, but most existing evaluations either chase frontier APIs or stop at accuracy on legacy benchmarks. This paper runs a uniform zero-shot evaluation of 41 open-weight models from 135M to 9B parameters across eight English single-label intent datasets, plus an auxiliary five-shot ATIS result, covering academic suites, a large voice-assistant corpus, and production e-commerce data. Beyond exact-match accuracy it measures calibration, typo and formatting robustness, ranking significance, Pareto efficiency, and benchmark saturation. The central practical claim is that instruction tuning outweighs moderate size jumps in this range: a 3B instruct model matches or beats several 7B base models, leading models are statistically hard to separate on MASSIVE alone, SNIPS is saturated and should be de-emphasized, and calibration effects of instruction tuning are mixed rather than uniformly bad. The point is actionable model-selection guidance for deployable open-weight intent classifiers.

Core claim

Within the evaluated open-weight sub-9B population, instruction tuning is a stronger signal than moderate parameter scale for zero-shot single-label intent classification: Qwen2.5-3B-Instruct reaches the same reported aggregate as a top-five 8B model and outperforms multiple 7B base models, all six cleanly matched base–instruct pairs improve with instruction tuning, differences among the top five on MASSIVE are pairwise non-significant under McNemar tests, and SNIPS is saturated (most models exceed 80% accuracy) while production-style sets still discriminate.

What carries the argument

A group-balanced eight-dataset zero-shot aggregate (standard IC composite of CLINC150/Banking77/SNIPS, plus MASSIVE, MTOP, and three HINT3 sets), scored by free-text exact-match accuracy under fixed greedy decoding, then extended with McNemar ranking tests, MASSIVE-scoped ECE/Brier calibration, single-seed input perturbations, and a Pareto front over accuracy, latency, and parameters.

Load-bearing premise

The paper treats sequence-level free-text log-probabilities on MASSIVE’s short labels as a usable stand-in for true label confidence, which is the basis for saying instruction tuning’s calibration effect is inconsistent rather than uniformly harmful.

What would settle it

Re-run the same 41 models with constrained decoding or full-test-set sampling and multi-seed typo perturbations: if matched base–instruct pairs no longer show consistent aggregate gains, top-five MASSIVE differences become significant, or SNIPS stops looking saturated relative to production sets, the selection guidance collapses.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • For maximum zero-shot accuracy in this range, prefer a strong 7B instruct model such as Mistral-7B-Instruct-v0.3 over larger base models alone.
  • Under tight memory or latency budgets, a 3B instruct model can be Pareto-rational and still competitive with much larger bases.
  • SNIPS should be demoted in modern IC leaderboards; MASSIVE, MTOP, Banking77, and HINT3-style production sets carry more ranking signal.
  • Single-benchmark rankings of leading open-weight models are unreliable without multi-dataset aggregates and significance tests.
  • Typo robustness and calibration must be checked per model; clean accuracy and instruction-tuning status do not reliably predict either.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Label-space design may matter as much as model choice: the cross-model play-music/music-query confusion suggests many “model errors” are actually ambiguous taxonomies that few-shot definitions or merged-label metrics could fix.
  • Reasoning-distilled models need deployment protocols with larger output budgets; fixed short-generation limits can zero out otherwise capable systems and skew comparative tables.
  • A natural next test is whether the same instruct-over-scale pattern holds for multi-label or multilingual intent under the same efficiency and robustness axes.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper presents a systematic zero-shot evaluation of 41 open-weight LLMs (15 families, 135M–9B) on eight English single-label intent-classification datasets, with ATIS reported separately as a five-shot auxiliary result. Beyond exact-match accuracy it reports confidence calibration (scoped to MASSIVE), robustness to input perturbations, McNemar tests and CIs on MASSIVE rankings, Pareto deployment efficiency, and benchmark saturation. Main claims are that instruction-tuned 3B models can outperform several 7B base models under the stated protocol, that top models on MASSIVE are pairwise statistically indistinguishable, that SNIPS is saturated (32/41 models >80%), and that instruction tuning’s effect on calibration is inconsistent rather than uniformly harmful. The work is framed as practical selection guidance for compute- and latency-constrained deployment.

Significance. If the reported rankings and tradeoffs hold under the stated free-text deployment protocol, the paper fills a genuine gap: prior intent benchmarks either target frontier proprietary models, use multiple-choice reformulations, or omit calibration, robustness, efficiency, and ranking reliability for the sub-9B open-weight range practitioners actually deploy. Strengths include breadth (41 models, production HINT3 sets), explicit statistical tests on MASSIVE, Pareto filtering with measured latency/VRAM, saturation counts that justify de-emphasizing SNIPS, and transparent limitations (ATIS subset skew, single perturbation seed, free-text calibration proxy). These make the resource useful as a menu of protocol-conditioned ranks even where causal claims about instruction tuning need tighter qualification.

major comments (3)
  1. [Results: Instruction Tuning vs. Parameter Scale; Finding-2; Evaluation Framework] Finding-2 (Instruction Tuning vs. Parameter Scale) and the abstract claim that instruction-tuned 3B models outperform 7B bases are measured only under free-text greedy generation, max 20 tokens, exact-match after normalization, with invalid outputs scored wrong (Evaluation Framework; Supplementary Answer-parsing logic). Instruct models are trained to obey “Reply with the label only”; base models are not. The paper’s own DeepSeek-R1-Distill-Qwen-1.5B near-zero score is attributed to reasoning tokens exhausting the budget, showing the protocol can zero models for non-IC reasons. Without per-model invalid/EMPTY_PRED rates (or a constrained-decoding / forced-choice control), a material share of the IT advantage may be format- and budget-compliance rather than intent separation. Please report invalid/empty rates by model (base vs instruct) and either (a) qualify Finding-2 as protocol-conditio
  2. [Confidence Calibration; Finding-8; Why Calibration is Restricted to MASSIVE] Calibration (RQ-8 / Finding-8) rests on sequence-level free-text logprobabilities on MASSIVE as a proxy for label-level confidence (Why Calibration is Restricted to MASSIVE; Limitations). The authors correctly flag length sensitivity and that low ECE can arise from consistent underconfidence. Given that the abstract and Finding-8 still state that instruction tuning’s calibration effect is “inconsistent rather than uniformly harmful,” the claim should be explicitly scoped in the abstract and finding box to “under approximate free-text sequence logprob confidence on MASSIVE,” and any base–instruct ECE comparison should note whether instruct models also change output-length distributions (which mechanically affect sequence logprobs).
  3. [Datasets; Limitations; Overall Model Rankings] Main accuracy uses deterministic first-500 index slices (complete HINT3 tests). Limitations documents a real ATIS majority-class skew (+3.8 pp for flight) that likely inflates ATIS figures, and states representativeness was not checked for the other five non-HINT3 sets. Because aggregate ranks and Finding-1 rest on these slices, either (i) report full-test-set scores for at least the top-10 models on CLINC150/Banking77/MASSIVE/MTOP, or (ii) quantify subset-vs-full agreement (e.g., rank correlation / accuracy delta) so readers can bound how much the 40-model ordering could move.
minor comments (6)
  1. [Robustness to Input Perturbations; Finding-7] Robustness (Finding-7) uses a single perturbation seed (seed 0). The text already cautions against over-generalizing Qwen2.5-7B-Instruct’s 1.3 pp typo drop; please also report at least 3 seeds or bootstrap CIs for the typo condition on the competitive subset so the “lowest observed degradation” claim is not a one-draw artifact.
  2. [Benchmark Saturation Analysis; Finding-3] Saturation is defined as “>50% of models exceed 80% accuracy.” The threshold is reasonable but free; a one-sentence sensitivity note (e.g., 70%/90% cutoffs) would show SNIPS remains saturated under nearby definitions.
  3. [Overall Model Rankings; Figure 1; Supplementary Table 2] Figure 1 and Table 2 report three-decimal aggregates; Qwen2.5-3B-Instruct and Llama-3.1-8B both show 0.632. State how ties are broken in rank order, or mark them as tied.
  4. [Supplementary Prompt Templates; Scoring] Prompt templates differ slightly across datasets (spaces vs underscores; Utterance/Intent vs Query/Label; uppercase for HINT3). A short note on whether normalization fully absorbs these surface differences would help reproducibility.
  5. [Related Work; Extended Related Work] Related Work cites IntentGrasp and Park et al. appropriately; when discussing saturation, a brief pointer to Larson & Leach (2022) already in the text is good—ensure the survey’s specific saturation observations are not overstated beyond what they claimed.
  6. [Figure 1 caption] Minor clarity: “group-balanced aggregate” is defined clearly in Evaluation Framework; consider repeating the six-group definition once in the Figure 1 caption so the figure is self-contained.

Circularity Check

0 steps flagged

No circularity: empirical bake-off reads rankings off held-out predictions under pre-specified prompts and metrics.

full rationale

This paper is a systematic zero-shot evaluation of 41 open-weight models on eight IC datasets (plus auxiliary 5-shot ATIS). Load-bearing claims—instruction tuning outweighing moderate scale within matched pairs, SNIPS saturation (32/41 >80%), non-significant top-5 McNemar tests on MASSIVE, Pareto efficiency, typo robustness, and inconsistent calibration effects—are empirical readouts from free-text greedy exact-match accuracy, ECE/Brier on MASSIVE logprobs, and single-seed perturbations. No parameter is fitted to a subset and then re-presented as a prediction of a closely related quantity; the group-balanced aggregate and the saturation threshold (>50% of models >80%) are explicit reporting definitions, not circular proofs, and a sensitivity check (Spearman ρ=0.995 vs simple mean) is reported. Citations are to external benchmarks and prior work (IntentGrasp, HINT3, etc.), not self-authored uniqueness theorems or ansatzes that force the results. There is no derivation chain that reduces by construction to its inputs. Protocol confounds (format compliance under free-text exact-match) are validity concerns, not circularity.

Axiom & Free-Parameter Ledger

6 free parameters · 6 axioms · 0 invented entities

Load-bearing structure is experimental protocol, not theoretical axioms. Findings rest on standard NLP evaluation assumptions plus several paper-specific design choices (subsetting, free-text confidence proxy, group-balanced mean, saturation threshold, fixed 20-token budget). No new physical entities; free parameters are threshold/design knobs rather than fitted scientific constants.

free parameters (6)
  • saturation_threshold = >50% of models >80% accuracy
    Saturation defined as >50% of models exceeding 80% accuracy; this cut drives the SNIPS de-emphasis recommendation.
  • group_balanced_aggregate_weights = equal weight on 6 groups (IC composite, MASSIVE, MTOP, 3×HINT3)
    Mean over six groups with CLINC150/Banking77/SNIPS collapsed to one composite so legacy benchmarks do not dominate; ranking claims use this weighting (sensitivity ρ=0.995 vs simple mean).
  • eval_subset_size = 500 (HINT3: full test sets)
    First 500 test indices for non-HINT3 datasets; affects all accuracy/CI/McNemar numbers.
  • max_output_tokens = 20
    Fixed 20-token generation budget; drives near-zero score for DeepSeek-R1-Distill-Qwen-1.5B via reasoning-trace truncation.
  • ECE_bins_and_confidence_proxy = n_bins=10; sequence-level logprob
    ECE with 10 equal-width bins on exp(sequence logprob) under free-text greedy decoding; calibration findings depend on this proxy.
  • perturbation_seed = seed 0
    Single realization seed 0 for typo/swap noise; typo-robustness ordering is conditioned on this draw.
axioms (6)
  • domain assumption Exact-match accuracy after lowercase/article/punctuation normalization is the primary quality metric for single-label IC.
    Stated in Scoring; all rankings flow from this metric family.
  • domain assumption Zero-shot free-text generation with a fixed label list in the prompt is a valid deployment-relevant protocol (no constrained decoding, no fine-tuning).
    Inference Protocol and Prompt Design; excludes constrained-decoding alternatives that would change calibration interpretability.
  • ad hoc to paper MASSIVE short-label generations align closely enough with the label vocabulary that sequence logprobabilities approximate answer confidence.
    Explicit scope justification for restricting ECE/Brier to MASSIVE; authors call it pragmatic and approximate.
  • standard math McNemar tests on n=500 MASSIVE items and Wald 95% CIs suffice to assess whether top-model ranking gaps are reliable on that benchmark.
    Statistical Significance of Rankings / Finding-4.
  • ad hoc to paper Index-ordered first-500 slices are adequate for cross-model comparison even if not random samples of the full test distribution.
    Datasets and Limitations; ATIS class-skew quantifies the risk.
  • domain assumption Pareto optimality over aggregate accuracy, latency, and parameter count identifies practically undominated deployment choices under the measured V100/vLLM regime.
    Deployment Efficiency section; hardware-specific latencies may not transfer.

reviewed 2026-07-31 · how reviews work

0 comments
Cite this review

Pith. "Pith review of Selecting Open-Weight Language Models for Zero-Shot Intent Classification: A Systematic Evaluation of 41 Models." pith.science (2026). https://pith.science/paper/Y2NTYWIE

@misc{pith2026260727421,
  author       = {Pith},
  title        = {Pith review of: Selecting Open-Weight Language Models for Zero-Shot Intent Classification: A Systematic Evaluation of 41 Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/Y2NTYWIE}},
  note         = {Machine review of arXiv:2607.27421}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Intent classification is a core component of task-oriented dialogue systems, yet practitioners have limited systematic guidance for selecting deployable open-weight language models under compute, latency, and robustness constraints. We present a systematic zero-shot evaluation of 41 open-weight language models spanning 15 families and the 135M--9B parameter range across eight English single-label intent-classification datasets. A ninth dataset, ATIS, uses five labeled demonstrations and is reported as an auxiliary five-shot result. The evaluation includes standard benchmarks, a large-scale voice-assistant corpus, and production-derived e-commerce datasets. Beyond exact-match accuracy, we analyze confidence calibration, robustness to realistic input perturbations, statistical reliability of model rankings, deployment efficiency, and benchmark saturation. Our results show that instruction-tuned 3B models can outperform several evaluated 7B base models, that differences among leading models on MASSIVE are statistically indistinguishable under pairwise McNemar tests, and that widely used benchmarks such as SNIPS have become saturated and no longer meaningfully discriminate among current open-weight models. Instruction tuning's effect on confidence calibration is inconsistent rather than uniformly harmful. These findings provide practical guidance for selecting and evaluating open-weight language models for intent classification.

Figures

Figures reproduced from arXiv: 2607.27421 by Cheryl Seals, Gerry Dozier, Parishruthi Ganesh.

Figure 1
Figure 1. Figure 1: Top 15 of the 40 comparable models ranked by group-balanced aggregate score over eight zero-shot datasets (ATIS excluded; see [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Parameter count versus aggregate score for the 40 comparable models (Qwen2.5-1.5B-Instruct omitted; 8-dataset zero-shot aggregate, [PITH_FULL_IMAGE:figures/full_fig_p012_2.png] view at source ↗
Figure 4
Figure 4. Figure 4: MASSIVE accuracy (n=500) for the top 14 models, with 95% confidence intervals (normal/Wald approximation, p ± 1.96p p(1 − p)/n). Point estimates and error bars are shown rather than bars from zero. The shaded band marks the top model’s CI range; the overlapping intervals illustrate that MASSIVE alone does not clearly distinguish several leading models (Finding-4). top-five models by MASSIVE accuracy, repor… view at source ↗
Figure 5
Figure 5. Figure 5: Prevalence of the play music→music query confusion across all 41 evaluated models on MASSIVE (500 samples): the number of models for which this pair is their single most common error (left bar) versus among their top-three most common errors (right bar). This is the dominant, cross-family, cross-scale confusion pattern discussed in the main paper (main paper’s Systematic Error Patterns subsection); both la… view at source ↗
Figure 6
Figure 6. Figure 6: Inference latency vs. 8-dataset zero-shot aggregate accuracy for the 40 standard-protocol models (Qwen2.5-1.5B-Instruct excluded). [PITH_FULL_IMAGE:figures/full_fig_p018_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Robustness to input perturbations on MASSIVE (500 samples). (a) Individual typo-induced accuracy drop, in percentage points, for [PITH_FULL_IMAGE:figures/full_fig_p020_7.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

43 extracted references · 20 linked inside Pith

  1. [1]

    A.; Awan, A

    Abdin, M.; Jacobs, S. A.; Awan, A. A.; Aneja, J.; Awadallah, A.; Awadalla, H.; Bach, N.; Bahree, A.; Bakhtiari, A.; Behl, H.; et al. 2024. Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone.arXiv preprint arXiv:2404.14219

    Pith/arXiv arXiv 2024

  2. [2]

    P.; Srivastav, V.; et al

    Allal,L.B.;Lozhkov,A.;Bakouch,E.;Blazquez,G.M.; Penedo, G.; Tunstall, L.; Marafioti, A.; Kydlicek, H.; Lajarin, A. P.; Srivastav, V.; et al. 2025. SmolLM2: WhenSmolGoesBig–Data-CentricTrainingofaSmall Language Model.arXiv preprint arXiv:2502.02737

    Pith/arXiv arXiv 2025

  3. [3]

    Bellagente, M.; Tow, J.; Mahan, D.; Phung, D.; Zhu- ravinskyi, M.; Adithyan, R.; Baicoianu, J.; Brooks, B.; Cooper, N.; Datta, A.; et al. 2024. Stable LM 2 1.6B Technical Report.arXiv preprint arXiv:2402.17834

    Pith/arXiv arXiv 2024

  4. [4]

    G.; Bradley, H.; O’Brien, K.; Hallahan, E.; Khan, M

    Biderman, S.; Schoelkopf, H.; Anthony, Q. G.; Bradley, H.; O’Brien, K.; Hallahan, E.; Khan, M. A.; Purohit, S.; Prashanth, U. S.; Raff, E.; Skowron, A.; Sutawika, L.; and van der Wal, O. 2023. Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling. InInternational Conference on Machine Learning, 2397–2430. PMLR

    2023

  5. [5]

    S.; Yvon, F.; et al

    BigScience Workshop; Le Scao, T.; Fan, A.; Akiki, C.; Pavlick, E.; Ilić, S.; Hesslow, D.; Castagné, R.; Luccioni, A. S.; Yvon, F.; et al. 2022. BLOOM: A 176B-Parameter Open-Access Multilingual Language Model.arXiv preprint arXiv:2211.05100

    Pith/arXiv arXiv 2022

  6. [6]

    D.; Dhariwal, P.; Neelakantan, A.; Shyam, P.; Sastry, G.; Askell, A.; et al

    Brown, T.; Mann, B.; Ryder, N.; Subbiah, M.; Kaplan, J. D.; Dhariwal, P.; Neelakantan, A.; Shyam, P.; Sastry, G.; Askell, A.; et al. 2020. Language models are few-shot learners. InAdvances in Neural Information Processing Systems, volume 33, 1877–1901

    2020

  7. [7]

    Cai, Z.; Cao, M.; Chen, H.; Chen, K.; Chen, K.; Chen, X.; Chen, X.; Chen, Z.; Chen, Z.; Chu, P.; et al

  8. [8]

    Casanueva, I.; Teˇmčinas, T.; Gerz, D.; Henderson, M.; and Vulić, I. 2020. Efficient intent detection with dual sentenceencoders. InProceedingsofthe2ndWorkshop on NLP for Conversational AI

    2020

  9. [9]

    Coucke, A.; Saade, A.; Ball, A.; Bluche, T.; Caulier, A.; Leroy, D.; Doumouro, C.; Gisselbrecht, T.; Calta- girone, F.; Lavril, T.; et al. 2018. Snips voice platform: an embedded spoken language understanding system for private-by-design voice interfaces.arXiv preprint arXiv:1805.10190

    Pith/arXiv arXiv 2018

  10. [10]

    DeepSeek-AI; Guo, D.; Yang, D.; Zhang, H.; Song, J.; Zhang, R.; Xu, R.; Zhu, Q.; Ma, S.; Wang, P.; et al. 2025. DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning.arXiv preprint arXiv:2501.12948

    Pith/arXiv arXiv 2025

  11. [11]

    FitzGerald, J.; Hench, C.; Peris, C.; Mackie, S.; Rottmann, K.; Sanchez, A.; Nash, A.; Urbach, L.; Kakarala, V.; Singh, R.; et al. 2022. MASSIVE: A 1M- example multilingual natural language understanding dataset with 51 typologically-diverse languages.arXiv preprint arXiv:2204.08582

    Pith/arXiv arXiv 2022

  12. [12]

    Grattafiori, A.; Dubey, A.; Jauhri, A.; Pandey, A.; Kadian, A.; Al-Dahle, A.; Letman, A.; Mathur, A.; Schelten, A.; Vaughan, A.; et al. 2024. The Llama 3 Herd of Models.arXiv preprint arXiv:2407.21783

    Pith/arXiv arXiv 2024

  13. [13]

    Guo, C.; Pleiss, G.; Sun, Y.; and Weinberger, K. Q

  14. [14]

    T.; Godfrey, J

    Hemphill, C. T.; Godfrey, J. J.; and Doddington, G. R

  15. [15]

    Hu, S.; Tu, Y.; Han, X.; He, C.; Cui, G.; Long, X.; Zheng, Z.; Fang, Y.; Huang, Y.; Zhao, W.; et al. 2024. MiniCPM: Unveiling the Potential of Small Language ModelswithScalableTrainingStrategies.arXivpreprint arXiv:2404.06395

    Pith/arXiv arXiv 2024

  16. [16]

    S.; de las Casas, D.; Bressand, F.; Lengyel, G.; Lample, G.; Saulnier, L.; et al

    Jiang,A.Q.;Sablayrolles,A.;Mensch,A.;Bamford,C.; Chaplot, D. S.; de las Casas, D.; Bressand, F.; Lengyel, G.; Lample, G.; Saulnier, L.; et al. 2023. Mistral 7B. arXiv preprint arXiv:2310.06825

    Pith/arXiv arXiv 2023

  17. [17]

    H.; Gonzalez, J

    Kwon, W.; Li, Z.; Zhuang, S.; Sheng, Y.; Zheng, L.; Yu, C. H.; Gonzalez, J. E.; Zhang, H.; and Stoica, I

  18. [18]

    Larson, S.; and Leach, K. 2022. A survey of intent classification and slot-filling datasets for task-oriented dialog.arXiv preprint arXiv:2207.13211

    Pith/arXiv arXiv 2022

  19. [19]

    J.; Clarke, C.; Lee, A.; Hill, P.; Kummerfeld, J

    Larson, S.; Mahendran, A.; Peper, J. J.; Clarke, C.; Lee, A.; Hill, P.; Kummerfeld, J. K.; Leach, K.; Laurenzano, M. A.; Tang, L.; et al. 2019. An evaluation dataset for intent classification and out-of-scope prediction. In Proceedings of EMNLP-IJCNLP

    2019

  20. [20]

    Li, X.; Lyu, T.; Yang, S.; et al. 2025. Consintbench: Evaluating language models on real-world consumer intent understanding.arXiv preprint arXiv:2510.13499

    arXiv 2025

  21. [21]

    Mehri, S.; and Eric, M. 2021. Example-driven intent prediction with observers. InProceedings of NAACL

    2021

  22. [22]

    HINT3: Raising the bar for intent detection in the wild

    Mehta,G.; Jain,C.;Chaturvedi,M.; andModi,K.2020. HINT3: Raising the bar for intent detection in the wild. InProceedings of the First Workshop on Insights from Negative Results in NLP

    2020

  23. [23]

    Park, G.; Baek, I.; Kim, B.; Shin, J.; and Lee, H

  24. [24]

    Qin, L.; Xu, X.; Che, W.; and Liu, T. 2020. AGIF: An adaptive graph-interactive framework for joint multiple intent detection and slot filling. InFindings of EMNLP

    2020

  25. [25]

    Schick, T.; and Schütze, H. 2021. Exploiting cloze- questions for few-shot text classification and natural language inference. InProceedings of EACL

    2021

  26. [26]

    Team GLM; Zeng, A.; Xu, B.; Wang, B.; Zhang, C.; Yin, D.; Zhang, D.; Rojas, D.; Feng, G.; Zhao, H.; et al

  27. [27]

    Wang, B.; and Komatsuzaki, A. 2021. GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model. https://github.com/kingoflolz/mesh-transformer-jax

    2021

  28. [28]

    Dynamic Label Name Refinement for Few- Shot Dialogue Intent Classification.arXiv preprint arXiv:2412.15603

    Pith/arXiv arXiv

  29. [29]

    Qwen2 Technical Report.arXiv preprint arXiv:2407.10671

    Yang, A.; Yang, B.; Hui, B.; Zheng, B.; Yu, B.; Zhou, C.;Li,C.;Li,C.;Liu,D.;Huang,F.;etal.2024. Qwen2 Technical Report.arXiv preprint arXiv:2407.10671

    Pith/arXiv arXiv 2024

  30. [30]

    Yang, A.; Yang, B.; Zhang, B.; Hui, B.; Zheng, B.; Yu, B.; Li, C.; Liu, D.; Huang, F.; et al. 2024. Qwen2.5 Technical Report.arXiv preprint arXiv:2412.15115

    Pith/arXiv arXiv 2024

  31. [31]

    Yang, Y.; Wang, W.; Xu, B.; et al. 2025. SessionIn- tentBench: A multi-task inter-session intention-shift modelingbenchmark.arXivpreprintarXiv:2507.20185

    Pith/arXiv arXiv 2025

  32. [32]

    ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Tools.arXiv preprint arXiv:2406.12793

    Pith/arXiv arXiv

  33. [33]

    Young, A.; Chen, B.; Li, C.; Huang, C.; Zhang, G.; Zhang, G.; Li, H.; Zhu, J.; Chen, J.; Chang, J.; et al

  34. [34]

    Y.; Guu, K.; Yu, A

    Wei, J.; Bosma, M.; Zhao, V. Y.; Guu, K.; Yu, A. W.; Lester, B.; Du, N.; Dai, A. M.; and Le, Q. V. 2022. Finetuned language models are zero-shot learners. In Proceedings of ICLR

    2022

  35. [35]

    Z.; Wallace, E.; Feng, S.; Klein, D.; and Singh, S

    Zhao, T. Z.; Wallace, E.; Feng, S.; Klein, D.; and Singh, S. 2021. Calibrate before use: Improving few-shot performance of language models.arXiv preprint. Supplementary Material This supplementary material provides complete model rank- ings, extended experimental results, prompt templates, cal- ibration results, robustness analyses, statistical tests, and...

    2021

  36. [38]

    Yin, Y.; Li, C.; and Carenini, G. 2026. Intentgrasp: A comprehensive benchmark for intent understanding. arXiv preprint arXiv:2605.06832

    Pith/arXiv arXiv 2026

  37. [40]

    Yi: Open Foundation Models by 01.AI.arXiv preprint arXiv:2403.04652

    Pith/arXiv arXiv

  38. [41]

    Zhang, P.; Zeng, G.; Wang, T.; and Lu, W. 2024. TinyLlama: An Open-Source Small Language Model. arXiv preprint arXiv:2401.02385

    Pith/arXiv arXiv 2024

  39. [1990]

    InSpeech and Natural Language: Proceedings of a Workshop

    The ATIS spoken language systems pilot corpus. InSpeech and Natural Language: Proceedings of a Workshop

  40. [2017]

    In Proceedings of the 34th International Conference on Machine Learning

    On calibration of modern neural networks. In Proceedings of the 34th International Conference on Machine Learning

  41. [2021]

    \n", "</s>

    (117 intents, hierarchical). HINT3 (Mehta et al. 2020) introducedthreerealIndiane-commerceproductiondatasets— Curekart, Powerplay11, Sofmattress—to expose the gap be- tween academic benchmarks and deployment reality. Larson and Leach (2022) surveyed IC datasets and noted growing saturation concerns, which our analysis confirms and extends. Zero-Shot IC wi...

    2020

  42. [2023]

    InProceedings of the 29th Symposium on Operating Systems Principles

    Efficientmemorymanagementforlargelanguage model serving with pagedattention. InProceedings of the 29th Symposium on Operating Systems Principles

  43. [2024]

    InternLM2 Technical Report.arXiv preprint arXiv:2403.17297

    Pith/arXiv arXiv

This paper was first reviewed by grok-4.5 on July 31, 2026.