In a Few Words: Comparing Weak Supervision and LLMs for Short Query Intent Classification

Arjen P. de Vries; Daria Alexander

REVIEW 3 major objections 4 minor 36 references

In a Few Words: Comparing Weak Supervision and LLMs for Short Query Intent Classification

T0 review · 3 major / 4 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read LLMs out-recall weak supervision but over-classify intents

desk verdict Useful, honest empirical study with a clean gold test, but the central recall claim may be confounded by ORCAS-I training labels generated with URL features. read the letter →

arxiv 2504.21398 v2 pith:THYM3F6G submitted 2025-04-30 cs.IR

Daria Alexander , Arjen P. de Vries This is my paper · ORCID

classification cs.IR

keywords userintentclassificationshortqueryweaksupervisionlargelanguagemodelsin-contextlearningweaklysupervisedfine-tuningORCAS-ILLaMA

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether large language models can replace weak supervision for classifying short web search queries into informational, navigational, and transactional intent. The authors compare LLaMA-3.1-8B and 70B with in-context learning, and 8B with weakly supervised fine-tuning, against the ORCAS-I weak supervision classifier on a 1,000-query manually labeled gold set. Their central finding is that every LLM configuration recovers more true intents than ORCAS-I (higher recall) but also produces more false positives (lower precision). The best LLM, LLaMA-3.1-8B fine-tuned on 45K random queries plus 15K high-confidence ones at threshold 0.88, reaches macro F1 0.769 versus ORCAS-I's 0.743, but with precision 0.743 versus 0.865, and it costs roughly 12 GPU-hours to train instead of 10 CPU-minutes. The authors conclude that ORCAS-I remains the better operational solution, while LLM recall could still be leveraged in a pipeline where weak supervision filters or reranks LLM candidates.

What carries the argument

The central objects are the ORCAS-I weak supervision classifier and the ORCAS-I-gold test set. ORCAS-I uses Snorkel labeling functions built from keywords, heuristics, and POS-tagging, with majority voting to label 10.4 million Bing queries as informational, navigational, or transactional; ORCAS-I-gold consists of 1,000 queries from that stream manually annotated by IR experts and held out from the weak labels. This gold set supplies the shared evaluation yardstick for all systems. On the LLM side the machinery is a prompt ladder for in-context learning (definitions only; definitions plus keywords; plus few-shot examples; plus clue-and-reasoning) and LoRA-based weakly supervised fine-tuning, where the authors augment random weak-label samples with self-labeled high-confidence queries at thresholds from 0.88 to 0.97.

What would settle it

Re-annotate a new random sample of, say, 5,000 ORCAS queries with independent expert labels and rerun the best in-context learning and weakly supervised fine-tuning configurations plus the ORCAS-I classifier on that sample; if the LLM's precision equals or exceeds ORCAS-I's on the new sample, or its recall advantage disappears, the paper's conclusion about LLMs struggling with precision would be overturned.

Watch

Extended reading notes

Core claim

On the ORCAS-I-gold test set, the ORCAS-I weak supervision classifier achieves macro precision 0.865, recall 0.701, F1 0.743 using only the query text. Every tested LLM configuration—both in-context learning with LLaMA-3.1-8B and 70B and LoRA fine-tuning of LLaMA-3.1-8B on weak labels—shows higher recall and lower precision, with the precision gap reported as statistically significant for the in-context learning runs. The best LLM result comes from weakly supervised fine-tuning on 45K randomly sampled ORCAS-I-2M queries supplemented by 15K queries whose labels the model itself assigned with confidence at or above 0.88; this configuration yields macro precision 0.743, recall 0.813, and F1 0.769. Because the weak supervision baseline can be trained and applied in about 10 CPU-minutes while the LLM pipeline requires close to 12 GPU-hours, the paper concludes that weak supervision is currently the better solution for operational intent classification, and that the LLM's recall advantage is best exploited as a candidate-generating front end to a weak-supervision filter.

Load-bearing premise

The comparison hinges on the 1,000-query ORCAS-I-gold test set being a representative, unbiased sample of short search queries, so that macro average precision and recall measured on it transfer to the full ORCAS population.

Editorial extensions

If this is right

For short query intent classification at the first taxonomy level, LLMs alone do not yet beat a weak-supervision classifier on the precision that an operational system needs.
Weakly supervised fine-tuning improves substantially over in-context learning, making it the more promising LLM direction.
Adding high-confidence model self-labels to the fine-tuning sample raises precision without sacrificing recall, with the best balance at threshold 0.88; raising the threshold beyond 0.90 over-filters and hurts precision.
A pipeline in which an LLM proposes broad candidate intents and ORCAS-I filters or reranks them could combine LLM recall with weak-supervision precision, if future work confirms it.
Cost is decisive at present: 12 GPU-hours for the best LLM versus 10 CPU-minutes for ORCAS-I, which keeps weak supervision preferable for large-scale deployment.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A testable extension: per-class confusion analysis on ORCAS-I-gold would show whether the LLM's precision loss comes from over-predicting one intent class, likely informational; if so, class-balanced reweighting or per-class thresholds could close much of the gap with no new data.
The confidence-threshold experiment treats the model's softmax probability as an accuracy signal; a calibrated confidence score, or an explicit abstain option, would likely push the precision-recall frontier further than raising the threshold did.
The cost comparison assumes training from scratch per deployment; amortized over repeated use, or with smaller adapters and quantization, the LLM's recall advantage may become more attractive than the headline 12-hours-versus-10-minutes figure suggests.
If the 1,000-query gold set under-represents ambiguous long-tail queries, the measured precision gap could go in either direction; replicating the comparison on a larger, independently annotated sample would show whether the ranking of systems is stable.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Quick take: this is a solid, honest short paper, but there's a confound in the training labels that could explain the main result. The paper compares LLaMA-3.1 ICL and weakly supervised fine-tuning against the ORCAS-I classifier on the 1,000-query ORCAS-I-gold set. What's new is the specific measurement: 8B and 70B ICL under four prompts, plus fine-tuning on 3K-60K samples with a high-confidence pseudo-label selection trick. The experimental protocol is generally careful: fixed gold test, paired permutation tests with Bonferroni correction, several sample sizes, and a cost analysis that honestly concludes ORCAS-I is still cheaper and better. That's worth credit.

The soft spot is a training-label mismatch. Section 3 says the ORCAS-I classifier uses clicked URLs for some categories, and Section 4 says the comparison is deliberately query-only because URLs are unavailable at query time. But the fine-tuning in 4.2 uses random samples drawn from ORCAS-I-2M, and I can't find anywhere that those 2M labels were produced by the query-only variant. If the released ORCAS-I-2M labels were generated with URL features, then the wSFT models were trained on a teacher with access to click information, while the baseline is denied it. The recall advantage the abstract highlights comes mainly from the fine-tuned models — the 70B ICL recall is not significantly above baseline. So the claim that LLMs outperform weak supervision in recall might be an artifact of the label mismatch, not the LLM's query-only semantic ability. This is testable: fine-tune on query-only-generated ORCAS-I labels and see if the recall gap holds. The authors should state which classifier version produced the 2M labels, or rerun the comparison.

Minor gaps: no decoding temperature for ICL, no explicit description of how raw model outputs were parsed into the three classes, and the high-confidence selection uses the model's own confidence, which is a known self-bias — though they report diminishing returns at higher thresholds, which mitigates it. These are secondary.

Who this is for: IR practitioners thinking about whether to swap a weak supervision classifier for an LLM, and anyone interested in the ORCAS-I benchmark. The cost comparison is useful. But until the confound is resolved, treat the headline recall claim with caution. I'd send it to peer review — the question is real and the fix is straightforward — but I'd require the label-version disclosure and a query-only fine-tuning run before accepting the interpretation.

Referee Report

3 major / 4 minor

Summary. The paper reports an empirical comparison of LLM-based approaches (in-context learning and weakly supervised fine-tuning with LLaMA-3.1-8B/70B-Instruct) against the ORCAS-I weak supervision classifier for short-query intent classification into informational, navigational, and transactional categories. Using the ORCAS-I-gold test set, the authors report macro-averaged precision, recall, and F1, finding that LLMs achieve higher recall but lower precision than the query-only ORCAS-I baseline. The best LLM configuration is LLaMA-3.1-8B-Instruct fine-tuned on 45K random plus 15K high-confidence queries from ORCAS-I-2M at threshold 0.88, reaching macro F1 0.769 versus the baseline's 0.743. The paper concludes that despite the LLMs' recall advantage, the ORCAS-I classifier remains the better solution in cost and effectiveness, and discusses a possible LLM-plus-weak-supervision pipeline.

Significance. If the empirical claims hold, the paper provides a useful benchmark for intent classification in IR: it gives clear evidence that in-context learning with 8B and 70B LLMs lags a well-designed weak supervision baseline, and that weakly supervised fine-tuning of an 8B model can improve over that baseline on recall and F1 while remaining worse on precision. The experimental protocol has notable strengths: a fixed externally annotated gold test set (ORCAS-I-gold), paired permutation tests with Bonferroni correction for the main ICL comparison, multiple training sample sizes, and an explicit cost comparison in GPU/CPU hours. The code release supports reproducibility. The main empirical finding is internally consistent with the tables, although one label-provenance question and a few smaller reporting gaps need to be addressed before the headline comparison can be fully trusted.

major comments (3)

[§3, §4.2] The wSFT models are trained on samples drawn from ORCAS-I-2M (§4.2), but the paper never states whether the labels in ORCAS-I-2M were produced by the query-only variant of the ORCAS-I classifier or by the full variant that uses clicked URLs for some categories (§3). The baseline row in Tables 1–2 is explicitly query-only (§5), so if ORCAS-I-2M labels used URL features, the training signal for the LLMs included post-click information that the baseline was denied. In that case the reported recall advantage of the wSFT configurations could be an artifact of the training-label mismatch rather than of query-only semantic ability. Please state the provenance of the ORCAS-I-2M labels; if they were produced with URL-enhanced labeling functions, re-run the wSFT comparison on query-only labels or explicitly re-frame the claim.
[Table 2 / §5.2.2] The claim that adding 15K high-confidence queries at threshold 0.88 yields the best configuration and improves over the 60K random sampling is based on differences of 0.013 in precision (0.743 vs. 0.730), 0.001 in recall, and 0.012 in F1. Unlike Table 1, Table 2 reports no paired permutation tests or other significance measures, so the 'best threshold' claim may be within noise. Either add significance testing for these comparisons or state explicitly that the differences are not statistically validated.
[§6] The conclusion that ORCAS-I 'offers the better solution, both in cost and effectiveness' is hard to reconcile with the reported numbers: the best LLM configuration has higher macro F1 (0.769 vs. 0.743) and recall (0.813 vs. 0.701), while ORCAS-I only wins on precision (0.865 vs. 0.743). The sentence should be qualified (e.g., in terms of precision per unit cost or for precision-critical settings) or revised to reflect the precision-recall trade-off.

minor comments (4)

[§4.1] The decoding parameters for LLaMA inference (temperature, top-p, max new tokens) are not reported; they can affect the distribution of generated labels. Please include them for reproducibility.
[§4.1] The procedure for extracting the predicted intent from model outputs is not described. Since the cited literature reports out-of-vocabulary labels, please specify how outputs were parsed (e.g., regex, string matching, or constrained decoding).
[§4.2] The text uses 'LoRa' where the acronym should be 'LoRA'. Also, the hyperparameter tuning description is vague ('dynamic hyperparameter tuning, decaying learning rate adaptively, increasing LoRA rank when performance remained low'); please list the final hyperparameters used for each run.
[References] Reference [20] is listed with 'Ben Mann, N Ryder, M Subbiah...' even though the cited paper is Brown et al.; reference [27] appears truncated. Please correct the bibliographic formatting.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: empirical comparison against an independent, manually annotated gold test set; self-citation of ORCAS-I is a baseline, not a derivation input.

full rationale

The paper is a purely empirical comparison between weak-supervision baselines (ORCAS-I) and LLM-based classifiers. The central claims (LLMs achieve higher recall but lower precision) are evaluated on the ORCAS-I-gold test set, which was manually annotated by IR experts on 1000 queries disjoint from the wSFT training data (Section 3: "These 1000 queries were annotated manually by IR experts"). Training data for wSFT consists of ORCAS-I-2M weak labels, and the gold test set is explicitly disjoint, so there is no reduction of a predicted quantity to its fitted input. The high-confidence query selection uses the model's own predictions to augment training data only, and the fixed external test set prevents this from becoming a circular prediction. Citation [1] (ORCAS-I) is prior work by the authors, but it functions as an external baseline to be outperformed rather than as a load-bearing justification of the paper's results; no uniqueness theorem or ansatz is imported via self-citation. Potential methodological concerns — e.g., whether ORCAS-I-2M labels used URL information while the baseline row is query-only, or that confidence thresholds were selected using test-set results — are threats to validity or external correctness, not circularity in the derivation chain. Therefore the manuscript exhibits no significant circularity.

Assumptions & free parameters 6 free parameters · 5 assumptions · 0 invented entities

The central claim is an empirical comparison, so the ledger lists the design hyperparameters and evaluation assumptions rather than physical parameters. No new entities are invented.

free parameters (6)

High-confidence selection thresholds = 0.88, 0.90, 0.95, 0.97
Used to select 15K pseudo-labeled queries for fine-tuning; 0.88 gave the best precision, others were worse. These are design hyperparameters, not fitted on the test set.
Learning rate = 2e-5, 1.5e-5
Hand-chosen learning rate for LoRA fine-tuning, reduced after 30K examples.
Batch size = 8, then 16
Hand-chosen training batch size with gradient accumulation step 8.
Epochs = 7
Training epochs for fine-tuning.
LoRA rank = not specified
The paper says 'increasing LoRA rank when performance remained low' but does not report the ranks used.
Few-shot examples per category = 5
Number of examples per class in ICL prompts.

assumptions (5)

domain assumption Broder's three-category intent taxonomy is a valid and sufficient framework for user intent classification.
Used throughout to define the classification task (Section 1, Section 3).
domain assumption ORCAS-I-gold is a reliable, representative gold standard.
Section 3: 1000 queries manually annotated by IR experts, not overlapping with training data; used for all evaluations.
standard math Permutation test with Bonferroni correction is appropriate for comparing classifiers on this test set.
Section 5.1: statistical significance testing across all model comparisons.
domain assumption Weak supervision labels from ORCAS-I-2M carry enough signal for fine-tuning a generalizable intent classifier.
Section 4.2: training data for wSFT; if labels were too noisy, the comparison would be less meaningful.
domain assumption LLM responses can be reliably parsed into one of the three labels.
Section 4.1: no parsing details given despite known out-of-vocabulary label risk cited in related work.

how reviews work

0 comments

Cite this review

Pith. "Pith review of In a Few Words: Comparing Weak Supervision and LLMs for Short Query Intent Classification." pith.science (2026). https://pith.science/paper/THYM3F6G

@misc{pith2026250421398,
  author       = {Pith},
  title        = {Pith review of: In a Few Words: Comparing Weak Supervision and LLMs for Short Query Intent Classification},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/THYM3F6G}},
  note         = {Machine review of arXiv:2504.21398}
}

read the original abstract

User intent classification is an important task in information retrieval. Previously, user intents were classified manually and automatically; the latter helped to avoid hand labelling of large datasets. Recent studies explored whether LLMs can reliably determine user intent. However, researchers have recognized the limitations of using generative LLMs for classification tasks. In this study, we empirically compare user intent classification into informational, navigational, and transactional categories, using weak supervision and LLMs. Specifically, we evaluate LLaMA-3.1-8B-Instruct and LLaMA-3.1-70B-Instruct for in-context learning and LLaMA-3.1-8B-Instruct for fine-tuning, comparing their performance to an established baseline classifier trained using weak supervision (ORCAS-I). Our results indicate that while LLMs outperform weak supervision in recall, they continue to struggle with precision, which shows the need for improved methods to balance both metrics effectively.

Figures

Figures reproduced from arXiv: 2504.21398 by the authors.

**Figure 2.** An example of misclassification by LLaMA-3.1-70B [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

**Figure 3.** Macro average scores for fine-tuning LLaMA-3.1- [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

36 extracted references · 18 canonical work pages

[1]

de Vries

Daria Alexander, Wojciech Kusa, and Arjen P. de Vries. 2022. ORCAS-I: queries annotated with intent using weak supervision. InProceedings of the 45th In- ternational ACM SIGIR Conference on Research and Development in Information Retrieval. 3057–3066

work page 2022
[2]

Valeriia Bolotova, Vladislav Blinov, Falk Scholer, W Bruce Croft, and Mark Sander- son. 2022. A non-factoid question-answering taxonomy. InProceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval. 1196–1207

work page 2022
[3]

Andrei Broder. 2002. A taxonomy of web search. InACM Sigir forum, Vol. 36. ACM New York, NY, USA, 3–10

2002
[4]

CB Chandrakala, Rohit Bhardwaj, and Chetana Pujari. 2024. An intent recognition pipeline for conversational AI.International Journal of Information Technology 16, 2 (2024), 731–743

work page 2024
[5]

Nick Craswell, Daniel Campos, Bhaskar Mitra, Emine Yilmaz, and Bodo Billerbeck
[6]

Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models.arXiv preprint arXiv:2407.21783(2024)

arXiv 2024
[7]

Mateusz Dubiel, Yasmine Barghouti, Kristina Kudryavtseva, and Luis A Leiva

work page
[8]

Alejandro Figueroa. 2015. Exploring effective features for recognizing the user intent behind web queries.Computers in Industry68 (2015), 162–169

work page 2015

Show all 36 references

[9]

X Han, W Zhao, N Ding, Z Liu, and M Sun Ptr. 2022. Prompt tuning with rules for text classification., 2022, 3.DOI: https://doi. org/10.1016/j. aiopen3 (2022), 182–192

2022 doi
[10]

Omar S Hemied and Hossam A Gabbar. 2024. RE-LLaMA: Large Language Models for Hydrogen Deployment: A Domain-Specific Approach in Renewable Energy. In 2024 8th International Symposium on Innovative Approaches in Smart Technologies (ISAS). IEEE, 1–4

2024
[11]

Bernard J Jansen, Danielle L Booth, and Amanda Spink. 2008. Determining the informational, navigational, and transactional intent of Web queries.Information Processing & Management44, 3 (2008), 1251–1266

2008
[12]

Bernard J Jansen, Amanda Spink, and Sherry Koshman. 2007. Web searcher interaction with the Dogpile. com metasearch engine.Journal of the American Society for Information Science and Technology58, 5 (2007), 744–755

2007
[13]

Hanlei Jin, Yang Zhang, Dan Meng, Jun Wang, and Jinghua Tan. 2024. A compre- hensive survey on process-oriented automatic text summarization with explo- ration of llm-based methods.arXiv preprint arXiv:2403.02901(2024)

2024
[14]

Ashish Kathuria, Bernard J Jansen, Carolyn Hafernik, and Amanda Spink. 2010. Classifying the user intent of web queries using k-means clustering.Internet Research20, 5 (2010), 563–581

2010
[15]

Junyi Li, Tianyi Tang, Wayne Xin Zhao, Jian-Yun Nie, and Ji-Rong Wen. 2024. Pre-trained language models for text generation: A survey.Comput. Surveys56, 9 (2024), 1–39

2024
[16]

Alisa Liu, Zhaofeng Wu, Julian Michael, Alane Suhr, Peter West, Alexander Koller, Swabha Swayamdipta, Noah A Smith, and Yejin Choi. 2023. We’re afraid language models aren’t modeling ambiguity.arXiv preprint arXiv:2304.14399(2023)

2023 arXiv
[17]

Haokun Liu, Derek Tam, Mohammed Muqeeth, Jay Mohta, Tenghao Huang, Mohit Bansal, and Colin A Raffel. 2022. Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning.Advances in Neural Information Processing Systems35 (2022), 1950–1965

2022
[18]

Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, and Weizhu Chen. 2021. What Makes Good In-Context Examples for GPT-3?arXiv preprint arXiv:2101.06804(2021)

2021 arXiv
[19]

Menglin Liu and Ge Shi. 2024. PoliPrompt: A High-Performance Cost-Effective LLM-Based Text Classification Framework for Political Science.arXiv preprint arXiv:2409.01466(2024)

2024 arXiv
[20]

Ben Mann, N Ryder, M Subbiah, J Kaplan, P Dhariwal, A Neelakantan, P Shyam, G Sastry, A Askell, S Agarwal, et al. 2020. Language models are few-shot learners. arXiv preprint arXiv:2005.141651 (2020)

2020 arXiv
[21]

Alaa Mohasseb, Mohamed Bader-El-Den, and Mihaela Cocea. 2019. A customised grammar framework for query classification.Expert Systems with Applications 135 (2019), 164–180

2019
[22]

Terufumi Morishita, Gaku Morio, Atsuki Yamaguchi, and Yasuhiro Sogawa. 2025. Enhancing Reasoning Capabilities of LLMs via Principled Synthetic Logic Corpus. Advances in Neural Information Processing Systems37 (2025), 73572–73604

2025
[23]

Marius Mosbach, Tiago Pimentel, Shauli Ravfogel, Dietrich Klakow, and Yanai Elazar. 2023. Few-shot fine-tuning vs. in-context learning: A fair comparison and evaluation.arXiv preprint arXiv:2305.16938(2023)

2023 arXiv
[24]

Greg Pass, Abdur Chowdhury, and Cayley Torgeson. 2006. A picture of search. InProceedings of the 1st international conference on Scalable information systems. 1–es

2006
[25]

Daniel E Rose and Danny Levinson. 2004. Understanding user goals in web search. InProceedings of the 13th international conference on World Wide Web. 13–19

2004
[26]

Hamidreza Rouzegar and Masoud Makrehchi. 2024. Enhancing Text Classification through LLM-Driven Active Learning and Human Annotation.arXiv preprint arXiv:2406.12114(2024)

2024 arXiv
[27]

D Russell et al . 2009. Task Behaviors During Web Search: The Difficulty of Assigning Labels∥, Proceedings of the 42nd Hawaii International Conference on System Sciences (HICSS). (2009)

2009
[28]

Xiaofei Sun, Xiaoya Li, Jiwei Li, Fei Wu, Shangwei Guo, Tianwei Zhang, and Guoyin Wang. 2023. Text classification via large language models.arXiv preprint arXiv:2305.08377(2023)

2023 arXiv
[29]

Yiming Tan, Dehai Min, Yu Li, Wenbo Li, Nan Hu, Yongrui Chen, and Guilin Qi
[30]

Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits reasoning in large language models.Advances in neural information processing systems35 (2022), 24824–24837

2022
[31]

Hanzi Xu, Renze Lou, Jiangshu Du, Vahid Mahzoon, Elmira Talebianaraki, Zhuoan Zhou, Elizabeth Garrison, Slobodan Vucetic, and Wenpeng Yin. 2024. LLMs’ Clas- sification Performance is Overclaimed.arXiv preprint arXiv:2406.16203(2024)

2024 arXiv
[32]

Vikas Yadav, Zheng Tang, and Vijay Srinivasan. 2024. Pag-llm: Paraphrase and aggregate with large language models for minimizing intent classification errors. InProceedings of the 47th international ACM SIGIR conference on research and development in information retrieval. 2569–2573

2024
[33]

Tong Zhang, Peixin Qin, Yang Deng, Chen Huang, Wenqiang Lei, Junhong Liu, Dingnan Jin, Hongru Liang, and Tat-Seng Chua. 2024. CLAMBER: A Benchmark of Identifying and Clarifying Ambiguous Information Needs in Large Language Models.arXiv preprint arXiv:2405.12063(2024)

2024 arXiv
[2020]

In Proceedings of the 29th ACM International Conference on Information & Knowledge Management

Orcas: 18 million clicked query-document pairs for analyzing search. In Proceedings of the 29th ACM International Conference on Information & Knowledge Management. 2983–2989
[2023]

InInternational Semantic Web Conference

Can ChatGPT replace traditional KBQA models? An in-depth analysis of the question answering performance of the GPT LLM family. InInternational Semantic Web Conference. Springer, 348–367
[2024]

On-device query intent prediction with lightweight LLMs to support ubiquitous conversations.Scientific Reports14, 1 (2024), 12731

2024

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.

[1] [1]

de Vries

Daria Alexander, Wojciech Kusa, and Arjen P. de Vries. 2022. ORCAS-I: queries annotated with intent using weak supervision. InProceedings of the 45th In- ternational ACM SIGIR Conference on Research and Development in Information Retrieval. 3057–3066

work page 2022

[2] [2]

Valeriia Bolotova, Vladislav Blinov, Falk Scholer, W Bruce Croft, and Mark Sander- son. 2022. A non-factoid question-answering taxonomy. InProceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval. 1196–1207

work page 2022

[3] [3]

Andrei Broder. 2002. A taxonomy of web search. InACM Sigir forum, Vol. 36. ACM New York, NY, USA, 3–10

2002

[4] [4]

CB Chandrakala, Rohit Bhardwaj, and Chetana Pujari. 2024. An intent recognition pipeline for conversational AI.International Journal of Information Technology 16, 2 (2024), 731–743

work page 2024

[5] [5]

Nick Craswell, Daniel Campos, Bhaskar Mitra, Emine Yilmaz, and Bodo Billerbeck

[6] [6]

Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models.arXiv preprint arXiv:2407.21783(2024)

arXiv 2024

[7] [7]

Mateusz Dubiel, Yasmine Barghouti, Kristina Kudryavtseva, and Luis A Leiva

work page

[8] [8]

Alejandro Figueroa. 2015. Exploring effective features for recognizing the user intent behind web queries.Computers in Industry68 (2015), 162–169

work page 2015