{"id":"f9e1363b-1db5-4c7d-9b1d-ffd792e5b03c","arxiv_id":"2506.03035","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Using BM25 lexical retrieval to select prompt examples improves few-shot slot-filling F1 scores on ATIS, SNIPS, SLURP, and MEDIA compared to random or intent-only selection.","lead":"This paper tests whether using information retrieval, specifically the lexical BM25 algorithm, to select the few examples shown to a large language model improves few-shot slot filling for spoken language understanding, and finds it does across four benchmarks. The result matters because it offers a simple, low-cost way to boost performance in low-resource SLU settings without lengthening the prompt.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"No error bars or multi-seed runs support the 'significant' improvement claim; single-run F1 differences in Table 2 may be within random-selection variance.","rationale":"The reader identified training-data contamination as the weakest assumption, but also listed 'lack of error bars' as an unresolved issue. I agree that contamination is a serious external-validity threat, yet the more load-bearing internal weakness is the absence of repeated runs and significance testing. Even if the model had never seen ATIS/SNIPS, the current evidence cannot distinguish a real retrieval benefit from random noise in example selection, because each configuration is evaluated once. This concern is concrete, testable with modest compute, and directly targets the paper's use of 'significantly.' The reader's verdict of CONDITIONAL remains appropriate: the paper should provide multi-seed results and significance tests before the central claim is accepted. Since my concern reinforces the same verdict, no change is needed.","tokens_in":8452,"tokens_out":3799,"duration_ms":49040,"concrete_test":"Run each example-selection method (Random, Intent-Based, BM25, BM25→Intent, Intent→BM25, ColBERT) with at least 10 different random seeds controlling example selection and prompt ordering, reporting mean F1 ± std on each dataset. Apply a paired bootstrap or Wilcoxon signed-rank test across test utterances for BM25 vs. Random and BM25 vs. Intent-Based, and also for BM25 vs. BM25→Intent and BM25 vs. Intent→BM25. If the BM25 advantage is not significant (p < 0.05) on at least the non-contaminated datasets (MEDIA and SLURP), the central claim of significant consistent improvement fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that BM25-based retrieval 'significantly enhances performance' over random and intent-based selection. However, Table 2 reports a single run per condition with no confidence intervals, standard deviations, or significance tests. The only stochastic component in the pipeline is the random example selection (for Random and Intent-Based baselines, and possibly the initial sampling for the deterministic retrievers if the training set is subsampled). With a single seed, the reported gaps between top methods are small: on SNIPS, BM25 achieves 84.38 vs. BM25→Intent at 85.21 and vs. Intent→BM25 at 84.32; on ATIS, BM25 achieves 86.63 vs. Intent→BM25 at 86.38. These differences are comparable to the variance one would expect from a different random draw of 10 examples per test utterance, especially on datasets like SLURP where the random baseline jumps from 15.20 to 43.87 depending on selection method. Figure 3 likewise shows curves without error bars, so claims of 'consistently outperforms' are not statistically anchored. This is distinct from the contamination issue: even on uncontaminated data, the claims require repeated sampling to establish that the observed ordering is not a fluke of one seed. Without this, the word 'significantly' in the abstract is unsupported, and the practical recommendation to replace random selection with BM25 rests on an unverified margin.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper addresses few-shot slot-filling in spoken language understanding by selecting prompt examples with information retrieval. The authors compare BM25, ColBERT, intent-based selection, and random selection, plus intent-filtered BM25 variants, on four benchmarks (ATIS, SNIPS, SLURP, MEDIA) using Hermes-3-Llama-3.1-8B. They report F1 scores, slot/intent presence rates, and scaling curves. The main claim is that lexical BM25 retrieval consistently improves over random and intent-based selection without lengthening the prompt. The authors also report a comparison to fine-tuned SOTA and acknowledge possible training-data contamination for ATIS and SNIPS.","tokens_in":8742,"tokens_out":6847,"duration_ms":76931,"significance":"If the claim holds, the contribution is practical and useful: a cheap, model-agnostic lexical retrieval step can improve few-shot SLU prompts, with no increase in prompt length. The paper includes a multi-benchmark comparison, a semantic retrieval baseline (ColBERT), an analysis of retrieved slot coverage, and a public code repository. The main weakness is that the headline results are presented without variance or significance testing, and the per-dataset choice of the best BM25 variant is made on the test set, so the magnitude and generalizability of the claimed improvement are not yet fully established.","major_comments":[{"comment":"The abstract and §4.1 describe the BM25 gains as 'significant' and 'consistent', but Table 2 reports a single F1 value per condition with no confidence intervals, standard deviations, or significance tests. Random selection is stochastic, and several comparisons are close (e.g., ATIS: BM25 86.63 vs. Intent→BM25 86.38; SNIPS: BM25 84.38 vs. BM25→Intent 85.21). Without repeated seeds or a paired test, the reader cannot distinguish a robust improvement from draw-to-draw variation. Please report means and standard deviations over at least 3–5 seeds, or bootstrap confidence intervals, and apply a paired significance test or an equivalence test for the key comparisons. This support is essential for the central claim.","section":"§4.1, Table 2, Abstract"},{"comment":"The authors correctly acknowledge that ATIS and SNIPS may overlap with the pretraining data of Hermes-3-Llama-3.1-8B. This is more than a caveat: if the model has memorized those datasets, the comparison on ATIS and SNIPS may reflect memorization rather than prompt example selection. Since the paper's cross-dataset claim relies partly on those two corpora, please provide additional evidence that the effect is not contamination-driven, for example by evaluating on a held-out subset not plausibly in pretraining, or by reporting results separately for datasets with and without contamination risk. The trend on SLURP and MEDIA is encouraging, but the current text does not quantify the extent of the problem.","section":"§4.2, §6"},{"comment":"The statement that the optimal configuration is dataset-dependent and the selection of the best BM25 variant per dataset are based on test-set F1. In a genuine few-shot deployment, a practitioner would not know which variant (BM25, BM25→Intent, Intent→BM25) is best. Please either provide a principled selection rule using a validation split or restrict the contribution to the family-level claim that all BM25 variants outperform random and intent-based selection. As written, the per-dataset 'best' configuration risks overfitting to the test set.","section":"§4.2, Table 2"},{"comment":"The scaling curves in Figure 3 have no error bars, and the claim that five BM25 examples always outperform one hundred random examples is not statistically anchored. Since the random baseline varies with the draw, the crossover point may be an artifact of one run. Please provide error bars, repeated runs, or the underlying per-shot data points for the scaling experiments.","section":"§4.3, Figure 3"}],"minor_comments":[{"comment":"The text refers to 'see Appendix in supplementary materials' for ColBERT details, but no appendix is present in the manuscript; please add the details or a pointer to the supplementary material.","section":"§3.3"},{"comment":"BM25 hyperparameters (k1, b) and the retrieval implementation are not specified; these are needed for reproducibility, especially since BM25 is the core method.","section":"§3.3"},{"comment":"The random seed and the number of random draws are not stated; without this, the stochastic baselines cannot be reproduced.","section":"§3.2, §4.1"},{"comment":"The conclusion says the method improves performance 'without additional computational cost', but §5 also states that 'the retrieval process can introduce additional computational overhead'; please reconcile these statements.","section":"§5"},{"comment":"The row label 'Fully Fine-tunned' contains a typo; it should read 'Fully Fine-tuned'.","section":"Table 2"}],"recommendation":"major_revision","confidential_remarks":"For the editor: the manuscript is within the journal's scope and the core idea is plausible. The main obstacles to acceptance are the absence of error bars/significance tests and the test-set-based configuration selection. The authors are transparent about limitations, which is good. I recommend major revision rather than rejection because the remaining issues can be addressed with additional experiments and text revisions."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe short take: this is a clean, practical empirical study that makes a plausible claim — BM25 retrieval for in-context examples beats random and intent-only selection for few-shot slot filling, and beats ColBERT — but it overstates confidence by using the word \"significant\" without any variance estimates.\n\nWhat's genuinely new: to my knowledge this is the first direct comparison of lexical vs semantic retrieval for example selection in SLU prompting. The finding that lexical beats semantic is non-obvious and worth publishing. The paper also does something rare: it explains the mechanism. Table 3 shows BM25 retrieves examples containing the target slots ~90% of the time, versus 75–83% for intent-based selection, which directly supports the performance gap. That's a nice analysis. The authors also release code and are transparent about the possible ATIS/SNIPS contamination.\n\nThe soft spots are real but narrower than the stress-test note suggests. The stress-test singles out the small gaps between BM25 and the two intent-interleaved variants on SNIPS/ATIS. True, those differences are within noise. But the headline comparison — BM25 vs random and vs intent-only — is not riding on a thin margin. On SLURP, BM25 gets 43.87 vs 15.20 random and 35.72 intent-based; on MEDIA, 55.56 vs 26.55 and 34.78. Those gaps are large enough to survive a few seeds. So the central recommendation stands despite the missing error bars.\n\nThe bigger issues: no multi-seed runs anywhere, and the abstract's \"significantly\" should be removed or supported. The optimal configuration (BM25 vs BM25→Intent vs Intent→BM25) appears dataset-dependent, and the paper admits they picked per dataset based on results — a mild form of test-set peeking that deserves a more prominent caveat. The contamination concern for ATIS/SNIPS is appropriately flagged in the Limitations section, though it does muddy the cross-dataset consistency claim.\n\nWho benefits: practitioners building few-shot SLU prompts and anyone researching prompt example selection. It's a solid workshop-to-conference paper that deserves a serious referee, not a desk reject. I'd ask for multi-seed runs or confidence intervals, a prespecified selection protocol, and a softer abstract claim. After those changes, I'd be comfortable citing it.","headline":"A practical, honest empirical study showing BM25 beats random and intent-based example selection for few-shot SLU prompts, but the gains are called 'significant' without variance estimates.","tokens_in":9238,"tokens_out":2461,"would_cite":true,"duration_ms":28818,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Selecting few-shot prompt examples with BM25 lexical retrieval improves slot-filling F1 by roughly 29 points on average over random selection across four spoken-language-understanding benchmarks.","keywords":["spoken language understanding","slot filling","few-shot learning","prompt engineering","information retrieval","BM25","in-context learning","large language models"],"falsifier":"Re-run the same BM25-versus-random comparison on a SLU dataset released after the model's training cutoff, or on paraphrased versions of ATIS and SNIPS that remove verbatim overlap with pretraining text; the central claim fails if the BM25 advantage disappears while random selection is unchanged.","tokens_in":8289,"feed_emoji":"🔍","tokens_out":9000,"duration_ms":87055,"temperature":0.7,"pith_summary":"This paper tries to establish that the choice of examples in a few-shot prompt is a first-order factor in spoken language understanding, and that a simple lexical retrieval step can make that choice well. Specifically, it claims that selecting the ten prompt examples by BM25 similarity to the current utterance improves slot-filling F1 by roughly 29 points on average over random selection across ATIS, SNIPS, SLURP, and MEDIA, without lengthening the prompt. The authors argue this matters because intent-based selection, the common prior choice, can miss utterances that share the same slots even when the intent differs. If the claim holds, few-shot SLU systems can replace random or intent-anchored example picking with an inexpensive, parameter-free retriever and get better annotations under the same prompt budget.","feed_headline":"Lexical retrieval lifts few-shot slot-filling F1 by 29 points","feed_subtitle":"Choosing prompt examples with BM25 beats random or intent-only picks on four SLU benchmarks, at no extra prompt length.","key_machinery":"The load-bearing object is BM25, a probabilistic lexical ranking function that scores each training utterance against the test utterance by word matching, with term-frequency saturation and document-length normalization. For each utterance to label, the pipeline retrieves the top-K most similar training utterances, formats them into a fixed prompt that lists the intent, slot definitions, and labeled examples, then asks an instruction-tuned 8-billion-parameter large language model to annotate the utterance. The comparison against ColBERT, a late-interaction neural retriever, isolates lexical overlap as the mechanism: BM25's advantage tracks its higher rate of retrieving examples that contain the expected slot types, not semantic relatedness.","core_discovery":"The paper's central claim is that lexical information retrieval, specifically BM25, is a more effective way to choose the ten examples in a few-shot slot-filling prompt than random selection, intent-based selection, or semantic retrieval with ColBERT. Across ATIS, SNIPS, SLURP, and MEDIA, BM25-selected prompts raise mean F1 from 38.43 (random) and 55.23 (intent-based) to 67.61, and combining BM25 with intent filtering does even better on the datasets with heavy slot overlap (69.31 for BM25-to-intent). The paper also reports that five BM25-chosen examples outperform one hundred randomly chosen ones, and that BM25 retrieves the expected slot types far more often (about 90% of prompts) than intent-based or random selection. The authors interpret this as evidence that lexical alignment between the utterance and the prompt examples is the most valuable signal to pass to the LLM.","pith_inferences":["The dataset-dependent ordering (plain BM25 on ATIS, intent-plus-BM25 on MEDIA and SLURP) suggests a cheap adaptive rule, such as measuring slot-overlap entropy per dataset and choosing whether to filter by intent before retrieval; the paper does not propose such a rule.","If the ATIS and SNIPS contamination the authors flag is real, the cleanest test of the mechanism is on fresh SLU data; until then the magnitude of the BM25 gain on those two datasets should be treated with caution.","The same retrieval-based example selection could transfer to other structured extraction tasks and other instruction-tuned models, but the paper's evidence is limited to one model and one slot-filling template.","A hybrid retriever that combines BM25's lexical precision with semantic embeddings might recover cases where the test utterance paraphrases slot vocabulary, an extension the paper's ColBERT results do not rule out."],"forward_implications":["Few-shot slot-filling prompts should retrieve examples by lexical similarity rather than picking randomly or by intent alone, since BM25 improves F1 on all four benchmarks tested.","The best configuration is dataset-dependent: plain BM25 wins on ATIS, while intent-filtered BM25 (in either order) wins on MEDIA and SLURP, where slots overlap across intents.","Prompt length is not the main lever: five BM25-chosen examples outperform 100 random examples, so shorter prompts can be both cheaper and more accurate.","Lexical alignment between the utterance and the examples matters more than semantic embedding similarity for this task, since ColBERT underperforms BM25 on every benchmark.","Retrieval quality is the bottleneck: when the retriever picks poor examples, as ColBERT does on SLURP, performance falls below intent-based selection."],"supporting_citations":[{"why":"Supplies the BM25 lexical ranking function that the proposed example-selection method is built on.","marker":"[20]"},{"why":"Provides the few-shot slot-filling prompt template and the 10-example setting that the proposed methods extend and compare against.","marker":"[16]"},{"why":"Motivates retrieval-based prompt example selection for few-shot learning, the core idea transferred here to SLU.","marker":"[19]"},{"why":"Defines the ColBERT late-interaction neural retriever used as the semantic retrieval baseline.","marker":"[21, 22]"},{"why":"Provides the base language model family underlying the instruction-tuned decoder used in all experiments.","marker":"[26]"},{"why":"Provides the instruction-tuned model variant actually used for slot-filling generation.","marker":"[27]"},{"why":"Defines the ATIS airline-information benchmark used in the evaluation.","marker":"[3]"},{"why":"Defines the SLURP voice-assistant benchmark used in the evaluation.","marker":"[5]"},{"why":"Defines the SNIPS personal-assistant benchmark used in the evaluation.","marker":"[23]"},{"why":"Defines the 2022 version of the French MEDIA hotel-reservation benchmark used in the evaluation.","marker":"[25]"}],"fun_headline_variants":["BM25 prompts lift few-shot SLU F1 by 29 points","Lexical retrieval beats intent for few-shot SLU prompts","IR-chosen examples boost few-shot SLU by 29 F1","BM25 beats random and intent in few-shot SLU","Lexical example selection wins in few-shot SLU"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the LLM has not already been trained on the evaluation corpora; the authors themselves flag in Sections 4.2 and 6 that ATIS and SNIPS may have leaked into pretraining data, and if that contamination is substantial the BM25-versus-random comparison would not measure genuine few-shot learning.","fun_headline_variants_meta":{"raw":{"variants":["BM25 prompts lift few-shot SLU F1 by 29 points","Lexical retrieval beats intent for few-shot SLU prompts","IR-chosen examples boost few-shot SLU by 29 F1","BM25 beats random and intent in few-shot SLU","Lexical example selection wins in few-shot SLU"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001455,"raw_usage":{"total_tokens":5829,"prompt_tokens":891,"completion_tokens":4938,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":507,"completion_tokens_details":{"reasoning_tokens":4852}},"tokens_in":507,"tokens_out":4938,"duration_ms":33156,"temperature":1.0,"reasoning_tokens":4852,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T11:09:07.818108+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the same BM25-versus-random comparison on a SLU dataset released after the model's training cutoff, or on paraphrased versions of ATIS and SNIPS that remove verbatim overlap with pretraining text; the central claim fails if the BM25 advantage disappears while random selection is unchanged.","supporting_citations":[{"cited_title":"BM25→Intent","cited_arxiv_id":null,"evidence_quote":"Provides the few-shot slot-filling prompt template and the 10-example setting that the proposed methods extend and compare against."},{"cited_title":"Multi-lingual Intent Detection and Slot Filling in a Joint BERT-based Model","cited_arxiv_id":"1907.02884","evidence_quote":"Motivates retrieval-based prompt example selection for few-shot learning, the core idea transferred here to SLU."},{"cited_title":"Croprompt: Cross-task interactive prompting for zero- shot spoken language understanding,","cited_arxiv_id":null,"evidence_quote":"Provides the base language model family underlying the instruction-tuned decoder used in all experiments."},{"cited_title":"Retrieval-based prompt selection for code-related few-shot learning,","cited_arxiv_id":null,"evidence_quote":"Provides the instruction-tuned model variant actually used for slot-filling generation."},{"cited_title":"We present the datasets used, describe the chosen language model, detail the retrieval mechanisms em- ployed, and specify the baseline and comparison methods","cited_arxiv_id":null,"evidence_quote":"Defines the ATIS airline-information benchmark used in the evaluation."},{"cited_title":"By integrating IR methods, specifically BM25, into the ex- ample selection process, we have addressed the challenges of overlapping intents and slots in complex datasets","cited_arxiv_id":null,"evidence_quote":"Defines the SLURP voice-assistant benchmark used in the evaluation."},{"cited_title":"Zero-shot spoken language understanding via large language models: A preliminary study,","cited_arxiv_id":null,"evidence_quote":"Defines the 2022 version of the French MEDIA hotel-reservation benchmark used in the evaluation."}],"review_version":1}