{"id":"30d81008-69c8-41e0-8ee7-553442221edf","arxiv_id":"2504.21398","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"low","formal_verification":"none","parameter_count":6,"one_line_summary":"LLMs outperform weak supervision on recall but not precision for short query intent classification, and fine-tuning beats in-context learning.","lead":"This paper compares a low-cost weak supervision classifier with large language models (LLaMA) for tagging search queries by user intent. It finds that LLMs catch more true intents (higher recall) but also tag many unrelated queries, so the cheaper weak supervision method still wins on precision and cost.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"wSFT may be trained on URL-enhanced ORCAS-I labels while the baseline is query-only; the reported recall advantage could be an artifact of this training-label mismatch.","rationale":"The reader's weakest assumption about the representativeness of ORCAS-I-gold is less compelling because the set is a random sample drawn from ORCAS, even if only 1,000 queries. The URL-label mismatch is a more direct threat to the fairness of the comparison that the entire paper is built on. The paper's own text supplies the ingredients: Section 3 says clicked URLs improve some ORCAS-I categories; Section 4 restricts the baseline to query-only retrieval; Section 4.2 fine-tunes on ORCAS-I-2M without specifying whether those labels came from the query-only or URL-enhanced variant. If ORCAS-I-2M labels used clicked URLs, the wSFT model was trained on a teacher with access to post-query information, while the ORCAS-I baseline in Tables 1–2 is deliberately query-only. That would make the recall comparison unequal and could explain part of the reported LLM recall advantage. The released code and ORCAS-I documentation should settle this quickly, so the paper remains conditional rather than being rejected: the authors need either to confirm that ORCAS-I-2M labels are query-only, or to re-run the fine-tuning with query-only labels. I therefore agree with the reader's conditional verdict, but for a different reason than the one stated in the reader's weakest_assumption.","tokens_in":8309,"tokens_out":11865,"duration_ms":133157,"concrete_test":"Inspect the ORCAS-I code/release or the original ORCAS-I paper (Alexander et al., 2022) to determine whether ORCAS-I-2M labels were generated with the full URL-enhanced labeling functions or a query-only variant. If the former, re-run the wSFT pipeline (Section 4.2) using labels from the query-only ORCAS-I variant on the same 45K random + 15K high-confidence samples and thresholds, and compare macro precision/recall to the ORCAS-I query-only baseline. If the recall advantage disappears, the central claim is confounded; if it persists, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3 describes ORCAS-I as a weak-supervision classifier whose labeling functions use keywords, heuristics, POS-tagging and, for some categories, clicked URLs. Section 4 says the paper deliberately compares only query-only retrieval, because clicked URLs are unavailable at query time, and the baseline row in Tables 1–2 is ORCAS-I query-only. What is not stated: whether the ORCAS-I-2M labels used for wSFT in Section 4.2 were produced by the query-only variant or by the full URL-enhanced variant. The released ORCAS-I-2M is described simply as the output of the ORCAS-I classifier. If those labels were generated with URL features, the wSFT models were trained on a teacher that saw post-query click information, while the baseline is denied it. The central 'LLMs outperform weak supervision in recall' claim is then not an apples-to-apples comparison: the recall advantage could come from the URL-informed training labels, not from the LLM's query-only semantic ability. This is a concrete, testable confound and should be resolved before accepting the claim.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper reports an empirical comparison of LLM-based approaches (in-context learning and weakly supervised fine-tuning with LLaMA-3.1-8B/70B-Instruct) against the ORCAS-I weak supervision classifier for short-query intent classification into informational, navigational, and transactional categories. Using the ORCAS-I-gold test set, the authors report macro-averaged precision, recall, and F1, finding that LLMs achieve higher recall but lower precision than the query-only ORCAS-I baseline. The best LLM configuration is LLaMA-3.1-8B-Instruct fine-tuned on 45K random plus 15K high-confidence queries from ORCAS-I-2M at threshold 0.88, reaching macro F1 0.769 versus the baseline's 0.743. The paper concludes that despite the LLMs' recall advantage, the ORCAS-I classifier remains the better solution in cost and effectiveness, and discusses a possible LLM-plus-weak-supervision pipeline.","tokens_in":8512,"tokens_out":5400,"duration_ms":54017,"significance":"If the empirical claims hold, the paper provides a useful benchmark for intent classification in IR: it gives clear evidence that in-context learning with 8B and 70B LLMs lags a well-designed weak supervision baseline, and that weakly supervised fine-tuning of an 8B model can improve over that baseline on recall and F1 while remaining worse on precision. The experimental protocol has notable strengths: a fixed externally annotated gold test set (ORCAS-I-gold), paired permutation tests with Bonferroni correction for the main ICL comparison, multiple training sample sizes, and an explicit cost comparison in GPU/CPU hours. The code release supports reproducibility. The main empirical finding is internally consistent with the tables, although one label-provenance question and a few smaller reporting gaps need to be addressed before the headline comparison can be fully trusted.","major_comments":[{"comment":"The wSFT models are trained on samples drawn from ORCAS-I-2M (§4.2), but the paper never states whether the labels in ORCAS-I-2M were produced by the query-only variant of the ORCAS-I classifier or by the full variant that uses clicked URLs for some categories (§3). The baseline row in Tables 1–2 is explicitly query-only (§5), so if ORCAS-I-2M labels used URL features, the training signal for the LLMs included post-click information that the baseline was denied. In that case the reported recall advantage of the wSFT configurations could be an artifact of the training-label mismatch rather than of query-only semantic ability. Please state the provenance of the ORCAS-I-2M labels; if they were produced with URL-enhanced labeling functions, re-run the wSFT comparison on query-only labels or explicitly re-frame the claim.","section":"§3, §4.2"},{"comment":"The claim that adding 15K high-confidence queries at threshold 0.88 yields the best configuration and improves over the 60K random sampling is based on differences of 0.013 in precision (0.743 vs. 0.730), 0.001 in recall, and 0.012 in F1. Unlike Table 1, Table 2 reports no paired permutation tests or other significance measures, so the 'best threshold' claim may be within noise. Either add significance testing for these comparisons or state explicitly that the differences are not statistically validated.","section":"Table 2 / §5.2.2"},{"comment":"The conclusion that ORCAS-I 'offers the better solution, both in cost and effectiveness' is hard to reconcile with the reported numbers: the best LLM configuration has higher macro F1 (0.769 vs. 0.743) and recall (0.813 vs. 0.701), while ORCAS-I only wins on precision (0.865 vs. 0.743). The sentence should be qualified (e.g., in terms of precision per unit cost or for precision-critical settings) or revised to reflect the precision-recall trade-off.","section":"§6"}],"minor_comments":[{"comment":"The decoding parameters for LLaMA inference (temperature, top-p, max new tokens) are not reported; they can affect the distribution of generated labels. Please include them for reproducibility.","section":"§4.1"},{"comment":"The procedure for extracting the predicted intent from model outputs is not described. Since the cited literature reports out-of-vocabulary labels, please specify how outputs were parsed (e.g., regex, string matching, or constrained decoding).","section":"§4.1"},{"comment":"The text uses 'LoRa' where the acronym should be 'LoRA'. Also, the hyperparameter tuning description is vague ('dynamic hyperparameter tuning, decaying learning rate adaptively, increasing LoRA rank when performance remained low'); please list the final hyperparameters used for each run.","section":"§4.2"},{"comment":"Reference [20] is listed with 'Ben Mann, N Ryder, M Subbiah...' even though the cited paper is Brown et al.; reference [27] appears truncated. Please correct the bibliographic formatting.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The main risk for acceptance is the unstated provenance of the ORCAS-I-2M labels: if those labels were generated with URL-enhanced labeling functions, the headline LLM-versus-weak-supervision comparison is not apples-to-apples. The authors should be asked to resolve this before publication. The paper uses the authors' own ORCAS-I as the baseline, which is not circular because the gold test set is manually annotated and fixed, but it does amplify the need for full transparency about the training labels."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a solid, honest short paper, but there's a confound in the training labels that could explain the main result. The paper compares LLaMA-3.1 ICL and weakly supervised fine-tuning against the ORCAS-I classifier on the 1,000-query ORCAS-I-gold set. What's new is the specific measurement: 8B and 70B ICL under four prompts, plus fine-tuning on 3K-60K samples with a high-confidence pseudo-label selection trick. The experimental protocol is generally careful: fixed gold test, paired permutation tests with Bonferroni correction, several sample sizes, and a cost analysis that honestly concludes ORCAS-I is still cheaper and better. That's worth credit.\n\nThe soft spot is a training-label mismatch. Section 3 says the ORCAS-I classifier uses clicked URLs for some categories, and Section 4 says the comparison is deliberately query-only because URLs are unavailable at query time. But the fine-tuning in 4.2 uses random samples drawn from ORCAS-I-2M, and I can't find anywhere that those 2M labels were produced by the query-only variant. If the released ORCAS-I-2M labels were generated with URL features, then the wSFT models were trained on a teacher with access to click information, while the baseline is denied it. The recall advantage the abstract highlights comes mainly from the fine-tuned models — the 70B ICL recall is not significantly above baseline. So the claim that LLMs outperform weak supervision in recall might be an artifact of the label mismatch, not the LLM's query-only semantic ability. This is testable: fine-tune on query-only-generated ORCAS-I labels and see if the recall gap holds. The authors should state which classifier version produced the 2M labels, or rerun the comparison.\n\nMinor gaps: no decoding temperature for ICL, no explicit description of how raw model outputs were parsed into the three classes, and the high-confidence selection uses the model's own confidence, which is a known self-bias — though they report diminishing returns at higher thresholds, which mitigates it. These are secondary.\n\nWho this is for: IR practitioners thinking about whether to swap a weak supervision classifier for an LLM, and anyone interested in the ORCAS-I benchmark. The cost comparison is useful. But until the confound is resolved, treat the headline recall claim with caution. I'd send it to peer review — the question is real and the fix is straightforward — but I'd require the label-version disclosure and a query-only fine-tuning run before accepting the interpretation.","headline":"Useful, honest empirical study with a clean gold test, but the central recall claim may be confounded by ORCAS-I training labels generated with URL features.","tokens_in":9036,"tokens_out":2495,"would_cite":false,"duration_ms":24134,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"LLMs out-recall weak supervision but over-classify intents","keywords":["user intent classification","short query classification","weak supervision","large language models","in-context learning","weakly supervised fine-tuning","ORCAS-I","LLaMA"],"falsifier":"Re-annotate a new random sample of, say, 5,000 ORCAS queries with independent expert labels and rerun the best in-context learning and weakly supervised fine-tuning configurations plus the ORCAS-I classifier on that sample; if the LLM's precision equals or exceeds ORCAS-I's on the new sample, or its recall advantage disappears, the paper's conclusion about LLMs struggling with precision would be overturned.","tokens_in":8103,"feed_emoji":"🎯","tokens_out":6567,"duration_ms":59697,"temperature":0.7,"pith_summary":"This paper asks whether large language models can replace weak supervision for classifying short web search queries into informational, navigational, and transactional intent. The authors compare LLaMA-3.1-8B and 70B with in-context learning, and 8B with weakly supervised fine-tuning, against the ORCAS-I weak supervision classifier on a 1,000-query manually labeled gold set. Their central finding is that every LLM configuration recovers more true intents than ORCAS-I (higher recall) but also produces more false positives (lower precision). The best LLM, LLaMA-3.1-8B fine-tuned on 45K random queries plus 15K high-confidence ones at threshold 0.88, reaches macro F1 0.769 versus ORCAS-I's 0.743, but with precision 0.743 versus 0.865, and it costs roughly 12 GPU-hours to train instead of 10 CPU-minutes. The authors conclude that ORCAS-I remains the better operational solution, while LLM recall could still be leveraged in a pipeline where weak supervision filters or reranks LLM candidates.","feed_headline":"LLMs out-recall weak supervision but over-classify intents","feed_subtitle":"Fine-tuned LLaMA-8B gets 0.769 F1 with high-confidence data, yet precision lags ORCAS-I: 0.743 vs 0.865.","key_machinery":"The central objects are the ORCAS-I weak supervision classifier and the ORCAS-I-gold test set. ORCAS-I uses Snorkel labeling functions built from keywords, heuristics, and POS-tagging, with majority voting to label 10.4 million Bing queries as informational, navigational, or transactional; ORCAS-I-gold consists of 1,000 queries from that stream manually annotated by IR experts and held out from the weak labels. This gold set supplies the shared evaluation yardstick for all systems. On the LLM side the machinery is a prompt ladder for in-context learning (definitions only; definitions plus keywords; plus few-shot examples; plus clue-and-reasoning) and LoRA-based weakly supervised fine-tuning, where the authors augment random weak-label samples with self-labeled high-confidence queries at thresholds from 0.88 to 0.97.","core_discovery":"On the ORCAS-I-gold test set, the ORCAS-I weak supervision classifier achieves macro precision 0.865, recall 0.701, F1 0.743 using only the query text. Every tested LLM configuration—both in-context learning with LLaMA-3.1-8B and 70B and LoRA fine-tuning of LLaMA-3.1-8B on weak labels—shows higher recall and lower precision, with the precision gap reported as statistically significant for the in-context learning runs. The best LLM result comes from weakly supervised fine-tuning on 45K randomly sampled ORCAS-I-2M queries supplemented by 15K queries whose labels the model itself assigned with confidence at or above 0.88; this configuration yields macro precision 0.743, recall 0.813, and F1 0.769. Because the weak supervision baseline can be trained and applied in about 10 CPU-minutes while the LLM pipeline requires close to 12 GPU-hours, the paper concludes that weak supervision is currently the better solution for operational intent classification, and that the LLM's recall advantage is best exploited as a candidate-generating front end to a weak-supervision filter.","pith_inferences":["A testable extension: per-class confusion analysis on ORCAS-I-gold would show whether the LLM's precision loss comes from over-predicting one intent class, likely informational; if so, class-balanced reweighting or per-class thresholds could close much of the gap with no new data.","The confidence-threshold experiment treats the model's softmax probability as an accuracy signal; a calibrated confidence score, or an explicit abstain option, would likely push the precision-recall frontier further than raising the threshold did.","The cost comparison assumes training from scratch per deployment; amortized over repeated use, or with smaller adapters and quantization, the LLM's recall advantage may become more attractive than the headline 12-hours-versus-10-minutes figure suggests.","If the 1,000-query gold set under-represents ambiguous long-tail queries, the measured precision gap could go in either direction; replicating the comparison on a larger, independently annotated sample would show whether the ranking of systems is stable."],"forward_implications":["For short query intent classification at the first taxonomy level, LLMs alone do not yet beat a weak-supervision classifier on the precision that an operational system needs.","Weakly supervised fine-tuning improves substantially over in-context learning, making it the more promising LLM direction.","Adding high-confidence model self-labels to the fine-tuning sample raises precision without sacrificing recall, with the best balance at threshold 0.88; raising the threshold beyond 0.90 over-filters and hurts precision.","A pipeline in which an LLM proposes broad candidate intents and ORCAS-I filters or reranks them could combine LLM recall with weak-supervision precision, if future work confirms it.","Cost is decisive at present: 12 GPU-hours for the best LLM versus 10 CPU-minutes for ORCAS-I, which keeps weak supervision preferable for large-scale deployment."],"supporting_citations":[{"why":"Supplies the weak-supervision baseline, the ORCAS-I-2M weak labels, and the ORCAS-I-gold test set used for evaluation.","marker":"[1]"},{"why":"Establishes the informational, navigational, and transactional intent taxonomy that defines the classification task.","marker":"[3]"},{"why":"Provides the ORCAS dataset of 10.4 million Bing queries from which training samples and gold test queries are drawn.","marker":"[5]"},{"why":"Motivates and supplies the few-shot in-context learning setup.","marker":"[20]"},{"why":"Supplies the clue-and-reasoning prompting strategy tested as the fourth in-context learning scenario.","marker":"[28]"},{"why":"Chain-of-thought prompting, which the clue-and-reasoning variant adapts to classification.","marker":"[30]"},{"why":"Supports the paper's assumption that LLM classification performance is overclaimed and motivates measuring precision and recall against a gold set.","marker":"[31]"},{"why":"Prior LLM intent classification work that the paper builds on and contrasts with for short query intent.","marker":"[32]"}],"fun_headline_variants":["LLMs out-recall weak supervision but lag on precision","Weak supervision wins precision, LLMs win recall","LLM recall edge overshadowed by weak supervision precision","LLM recall boost suggests candidate-generator role","For intent classification, weak supervision beats LLM precision"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The comparison hinges on the 1,000-query ORCAS-I-gold test set being a representative, unbiased sample of short search queries, so that macro average precision and recall measured on it transfer to the full ORCAS population.","fun_headline_variants_meta":{"raw":{"variants":["LLMs out-recall weak supervision but lag on precision","Weak supervision wins precision, LLMs win recall","LLM recall edge overshadowed by weak supervision precision","LLM recall boost suggests candidate-generator role","For intent classification, weak supervision beats LLM precision"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000723,"raw_usage":{"total_tokens":3258,"prompt_tokens":976,"completion_tokens":2282,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":592,"completion_tokens_details":{"reasoning_tokens":2207}},"tokens_in":592,"tokens_out":2282,"duration_ms":16886,"temperature":1.0,"reasoning_tokens":2207,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T05:04:05.498714+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-annotate a new random sample of, say, 5,000 ORCAS queries with independent expert labels and rerun the best in-context learning and weakly supervised fine-tuning configurations plus the ORCAS-I classifier on that sample; if the LLM's precision equals or exceeds ORCAS-I's on the new sample, or its recall advantage disappears, the paper's conclusion about LLMs struggling with precision would be overturned.","supporting_citations":[{"cited_title":"de Vries","cited_arxiv_id":null,"evidence_quote":"Supplies the weak-supervision baseline, the ORCAS-I-2M weak labels, and the ORCAS-I-gold test set used for evaluation."},{"cited_title":"LLMs' Classification Performance is Overclaimed","cited_arxiv_id":"2406.16203","evidence_quote":"Supports the paper's assumption that LLM classification performance is overclaimed and motivates measuring precision and recall against a gold set."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Prior LLM intent classification work that the paper builds on and contrasts with for short query intent."}],"review_version":1}