{"id":"2af7776e-75fc-49e3-8b5f-7f1200cd1418","arxiv_id":"2505.18241","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A k-nearest-neighbor search over multilingual query embeddings provides zero-shot intent classification for low-resource languages, with accuracy below translation-based and supervised baselines.","lead":"This paper tests a simple idea: classify a user's query by looking up the most similar labeled queries instead of training a classifier. It finds the approach works moderately for low-resource languages with no target-language training data, but trails translation-based and supervised baselines.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The zero-shot claim rests on cross-lingual alignment of off-the-shelf encoders, but Table 4 shows XLM-R collapses to 0.119 accuracy on Swahili while LaBSE reaches 0.486; without validating the retrieval mechanism, 'reasonable' is not established as a general property.","rationale":"The reader's weakest assumption identifies exactly the load-bearing issue: the method presupposes that multilingual encoders create a shared space where neighbor intent labels transfer across languages. The paper's own Table 4 provides a stark counterexample in XLM-R on Swahili, while the text in Section 4.3 overstates the result by claiming the approach 'can excel.' My stress-test does not change the conditional verdict: for LaBSE on these languages, the method is a feasible feasibility data point, but the general 'reasonable zero-shot performance' claim is not robust across the two tested encoders, and the retrieval mechanism itself is never directly measured. The proposed concrete test would settle whether the failure is in embedding alignment or in the label-resolution step, and would clarify the boundary of the central claim. No independent evidence (code, error bars, or ablations) is provided, but the paper is honest about its baselines and limitations, so conditional acceptance with a request for this diagnostic is appropriate.","tokens_in":8151,"tokens_out":5645,"duration_ms":47279,"concrete_test":"Run a retrieval-diagnostic on the Section 4.3 setup: for each Swahili test query, retrieve top-31 neighbors from the high-resource index using both LaBSE and XLM-R, and compute (i) the fraction of retrieved neighbors whose true intent matches the query's intent (retrieval hit-rate) and (ii) the accuracy of a classifier that uses only the top-1 neighbor's label. If XLM-R's hit-rate is low and top-1 accuracy is near 0.119, the failure is localized in cross-lingual embedding alignment; if hit-rate is high but majority-vote accuracy is low, the failure is in label resolution. This isolates where the load-bearing assumption breaks.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim of zero-shot low-resource intent classification depends on the assumption that off-the-shelf multilingual encoders align queries from different languages so that nearest neighbors in the index share intent labels. Section 3.4 and Table 4 show this assumption is not generally satisfied: for Swahili, XLM-RoBERTa Sim-Search achieves only 0.119 accuracy, versus 0.486 with LaBSE, and both are far below the 0.70 translation baseline; for Urdu and Indonesian, XLM-R is near 0.5. The paper never directly validates the retrieval mechanism, e.g., how often the top-k retrieved neighbors actually have the same intent as the low-resource query. Without such validation, or a criterion for choosing an encoder, the abstract's 'reasonable performance' is not established as a property of the proposed method; it is an artifact of a favorable encoder-language match. The Section 4.3 statement that the approach 'can excel' is contradicted by the XLM-R Swahili result. The reader's condition is therefore appropriate, but the underlying alignment assumption should be tested directly before accepting the claim.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a zero-shot intent classification method for low-resource languages: instead of training a classifier, the authors index labelled queries from high-resource languages using off-the-shelf multilingual sentence encoders (LaBSE, XLM-R, RoBERTa, Sentence-T5), and then label a new query by majority vote over its k nearest neighbors in this index. They evaluate the method on English datasets (ATIS, HINT3, CLINC-150), multilingual datasets (MultiATIS++, MASSIVE), and a zero-shot low-resource subset of MASSIVE (Swahili, Urdu, Indonesian). The central claim is that this retrieval-based formulation achieves 'reasonable' zero-shot accuracy for low-resource language queries without any target-language annotations, retraining, or translation services, while being quick to deploy.","tokens_in":8284,"tokens_out":4215,"duration_ms":38350,"significance":"If the central claim holds, the paper offers a practical and lightweight recipe for bootstrapping intent classification in low-resource languages, which is a real need in the IR industry. The method is simple, reuses off-the-shelf encoders, and requires no training or translation dependency, so a reproducible implementation could be a useful baseline for subsequent work. The paper also explicitly evaluates the approach against translation-based and supervised alternatives, which is valuable. However, the significance is limited by the mixed results: for some language-encoder pairs the accuracy is near-random, and the 'reasonable' claim is not established as a general property of the method. The paper would be strengthened by directly validating the retrieval mechanism and by characterizing when the method succeeds.","major_comments":[{"comment":"The hyperparameter k is selected by a grid search on the ATIS test set (Section 2.2: 'Using the test set in ATIS, we perform a grid search') and then reused for all subsequent experiments. Tuning on the test set, without a validation split, inflates the reported performance and makes the low-resource numbers conditional on a test-set-derived choice. In addition, HINT3 uses a different k (k=5) chosen in an ad hoc way. The authors should tune k on a training or validation portion of ATIS, report the sensitivity of the low-resource results to k, and specify the tuning protocol for HINT3.","section":"§2.2, §4"},{"comment":"The paper conflates N (the number of samples indexed per class) with k (the number of nearest neighbors). Section 3.2 states 'we choose N = 31 for our C-way N-shot sample set. The choice of N = 31 is to parallel our findings from section 2.2 that indexing k = 31 was optimal.' These are different hyperparameters, and the justification is unclear. For HINT3, the text says k is lowered to 5 because of class sparsity, but it does not say whether N is also changed. The reported accuracy could depend as much on N as on the retrieval mechanism, so the authors must report N and k separately for every dataset and clarify the relationship between them.","section":"§3.2, §4.1"},{"comment":"The conclusion in Section 4.3 that 'the proposed intent classification approach can excel in a low-resource, zero-shot intent classification scenario' is contradicted by the XLM-RoBERTa result on Swahili: 0.119 accuracy in the high-resource-index setting, which is near the random baseline for MASSIVE. The success of the method depends strongly on the encoder-language pair. Because the zero-shot premise rests on cross-lingual alignment of off-the-shelf encoders, the paper should directly validate the retrieval mechanism—for example, report the agreement rate between the top-k retrieved neighbors and the true intent for each language—and provide guidance for choosing an encoder for a given target language. Without such validation, 'reasonable performance' is an artifact of a favorable encoder-language match for some settings, not a property of the method.","section":"§4.3, Tables 3–4"},{"comment":"All results are reported as single runs with no error bars, seeds, or significance tests. Given that kNN majority voting can produce ties and that the differences between some comparisons are small (e.g., LaBSE high-resource vs. all-w/o-target for id-ID: 0.538 vs. 0.519), the reader cannot assess whether the reported differences are meaningful. The authors should run multiple seeds or bootstrap resamples and report means and standard deviations, at least for the main low-resource results.","section":"§4, Tables 1–4"}],"minor_comments":[{"comment":"There is a typo in the reference to Khalil et al.: 'Engish' should be 'English'.","section":"References"},{"comment":"The spelling of 'LaBSE' is inconsistent: both 'LaBSE' and 'LabSE' appear in the text and tables. Please make it consistent.","section":"Throughout"},{"comment":"Figure 1 reports average accuracy and F1 across five encoder models, but no details are given about which five encoders were used, whether the evaluation was on the ATIS test set, and whether error bars or variance across encoders are shown. Since this figure is the sole basis for choosing k=31, it needs more detail.","section":"§2.2, Figure 1"},{"comment":"The literature numbers in Table 1 are not all comparable: for HINT3, the literature row reports only accuracy (0.75) and F1 is N/A. The text should clarify which models produced these numbers and whether they were trained on the full training set, as is already stated for ATIS and CLINC-150.","section":"§4.1, Table 1"},{"comment":"The algorithm does not specify whether the embeddings are L2-normalized before cosine similarity computation. If FAISS inner product is used, normalization changes the results; please state the exact embedding post-processing.","section":"Algorithm 1, §2.1"},{"comment":"The phrase 'under this realistic setting' is used to describe the high-resource-only index, but the paper also reports results for the 'all w/o target' setting in Table 3. The two settings should be more clearly distinguished in the prose, and the authors should emphasize that the high-resource-only setting is the one that matches the zero-shot claim.","section":"§4.3"},{"comment":"The abstract's claim of 'reasonable intent classification performance' is too broad given the near-random result for Swahili with XLM-RoBERTa. The authors should either qualify the claim (e.g., 'for several language-encoder combinations') or provide the stated validation that would make the claim general.","section":"Abstract, §5.1"}],"recommendation":"major_revision","confidential_remarks":"The paper is an empirical evaluation and the central recipe is easy to state, so the main work after revision is to close the experimental gaps rather than to redo the theory. The most serious issue for the journal is the use of the ATIS test set for hyperparameter selection; if the authors address that, the retrieval validation, and the overclaiming in Section 4.3, the paper could become a useful contribution."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper tests a simple idea: treat intent classification as k-nearest neighbor search over pretrained multilingual embeddings, with no fine-tuning and no target-language data. It's not a new method—Bari et al. already used cross-lingual nearest neighbors—but this is the first systematic zero-shot evaluation on low-resource languages in MASSIVE, and the comparison against classification heads and a translation pipeline is solid and honest. The main result is that LaBSE embeddings give about 0.49–0.54 accuracy on Swahili, Urdu, and Indonesian, which is far above random and not embarrassing as a zero-shot baseline. That is a useful feasibility data point.\n\nThe biggest soft spot: the claim that the approach \"can excel\" (Section 4.3) is contradicted by the XLM-R Swahili result of 0.119 accuracy, which is essentially random. The paper never validates the retrieval mechanism directly, so we don't know when neighbors are actually semantically aligned. The reader's stress-test is right: \"reasonable performance\" is not a property of the method, it's a property of a favorable encoder-language match. The abstract should say that.\n\nMinor issues: k is tuned on the ATIS test set, which is a leak of sorts; HINT3 uses a different k without explanation; no error bars or seeds are reported; no code is provided. None of these are fatal in a short empirical paper, but they cap the confidence level.\n\nThe paper is well written and the experiments are reproducible in principle. I would send it to a serious referee, but I'd expect a request for closer examination of the encoder-dependence and a softer conclusion. It's more of a workshop-quality contribution than a major result, but it is a genuine data point for practitioners who need a no-training baseline.","headline":"Useful feasibility study of a known kNN recipe for zero-shot intent classification, but the 'reasonable' claim holds only for some encoder-language pairs and the paper overstates one result.","tokens_in":8922,"tokens_out":2174,"would_cite":true,"duration_ms":18195,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Casting intent classification as similarity search lets a query index classify low-resource languages without target-language labels or retraining.","keywords":["intent classification","query similarity search","k-nearest neighbors","multilingual sentence embeddings","zero-shot transfer","low-resource languages","LaBSE","XLM-R"],"falsifier":"Measure, on a large multilingual intent dataset, the probability that a query's nearest neighbor in a high-resource-language-only index has the same intent; if for a given language-encoder pair this probability is at chance level, the proposed zero-shot transfer is falsified for that pair, and this can be checked directly from the MASSIVE data used in the paper.","tokens_in":7828,"feed_emoji":"🔍","tokens_out":8131,"duration_ms":75063,"temperature":0.7,"pith_summary":"This paper argues that intent classification can be treated not as a trained classifier but as a nearest-neighbor lookup: label a new query by the majority intent of the most similar already-labeled queries in a multilingual embedding space. The intended payoff is zero-shot support for low-resource languages: a system with labeled queries in English, Chinese, Spanish, French, and Japanese can classify queries in Swahili, Urdu, or Indonesian without any target-language annotations and without calling a machine translation service. The paper demonstrates the idea on English, multilingual, and low-resource benchmarks, and reports that an index built only from high-resource languages performs about as well as an index built from all available languages except the target. The approach does not beat supervised classifiers or translation-based pipelines in accuracy, but it is immediately deployable, requires no retraining or re-indexing, and works when no translation model exists for the target language.","feed_headline":"Nearest-neighbor lookup labels low-resource intents with no training","feed_subtitle":"Classify Swahili, Urdu, or Indonesian queries using only high-resource-language labels.","key_machinery":"The load-bearing object is the query index. Each labeled training query is embedded with a pretrained sentence encoder; the embeddings are stored in an approximate nearest-neighbor index; at inference, the encoder embeds the new query, a search returns the top-$k$ closest indexed queries under cosine similarity, and the predicted intent is the mode of their labels. A grid search on ATIS over $k$ from 1 to 75 across five encoders fixes $k=31$, and experiments then index $N=31$ queries per class per language, with English-only, multilingual, and high-resource-only variants. The mechanism transfers intent labels across languages only insofar as the encoder places paraphrases of the same intent close together regardless of language.","core_discovery":"The central claim is that casting query intent classification as a similarity search problem—indexing labeled queries as embeddings and taking a majority vote over the top-$k$ approximate nearest neighbors of an incoming query—yields usable zero-shot intent classification for low-resource languages. Using off-the-shelf multilingual sentence encoders (LaBSE and XLM-R) and an approximate nearest-neighbor index, the authors show that when the query index contains labels only from five high-resource languages, accuracy on Swahili, Urdu, and Indonesian test queries is comparable to an index that includes labeled queries from every language except the target. This is their evidence that the approach can be dropped into a system with no annotated data from the target language and no translation dependency.","pith_inferences":["Beyond the paper's experiments, the severe Swahili failure with XLM-R suggests a cheap pre-deployment diagnostic: for each new language, measure neighbor-label agreement between that language's queries and the high-resource index before committing to zero-shot transfer, and route low-agreement languages to a translation baseline or a small annotated support set.","The optimal $k=31$ found on ATIS is unlikely to transfer universally; the right $k$ should grow with index size and class distribution, so production systems should tune it on a held-out set in any available language rather than adopt 31 as a constant.","The paper's framing suggests a scaling law worth testing: if the index is enlarged with more balanced queries per class, the gap between similarity search and a supervised classification head on full data should shrink, especially on balanced datasets like MASSIVE."],"forward_implications":["A service that already has labeled queries in a handful of high-resource languages can add a new low-resource language by embedding the incoming queries and querying the existing index, with no training run and no target-language labels.","Because the index needs no translation step, the method remains viable for low-resource languages that are not covered by off-the-shelf machine translation models.","The parity between the all-languages-except-target index and the five-high-resource-languages-only index implies that adding more languages to the index is not what drives zero-shot performance; the encoder's cross-lingual alignment is the active ingredient.","In the English and multilingual experiments, the similarity-search classifier uses only a small subsample of the labeled data and stays within a few points of a classification head trained on the same or larger data, suggesting the method is data-efficient as well as deployment-efficient."],"supporting_citations":[{"why":"Supplies the 50-language MASSIVE dataset, including the Swahili, Urdu, and Indonesian test sets used for the zero-shot low-resource evaluation.","marker":"[10]"},{"why":"LaBSE is the multilingual sentence encoder whose embeddings define the cross-lingual similarity space for one set of experiments.","marker":"[9]"},{"why":"XLM-R is the multilingual encoder used for the other set of experiments, including the failure case in Swahili.","marker":"[4]"},{"why":"Supplies the approximate nearest-neighbor index implementation used to make the similarity search efficient.","marker":"[8]"},{"why":"This prior nearest-neighbor cross-lingual work is the direct comparison point; the paper's method differs by not fine-tuning and requiring zero target-language examples.","marker":"[3]"},{"why":"MultiATIS++ supplies the multilingual ATIS benchmark used in the multilingual experiments.","marker":"[18]"},{"why":"The ATIS dataset is used to grid-search the optimal $k$ for majority-vote label resolution.","marker":"[11]"},{"why":"NLLB-200 supplies the machine-translation baseline that the proposed approach aims to be independent of.","marker":"[5]"}],"fun_headline_variants":["Zero-shot intents via similarity search on low-resource queries","Similarity search labels low-resource intents with no training data","No training needed: classify low-resource intents via nearest neighbors","Query similarity outsources intent labels for low-resource languages","Nearest-neighbor intent detection works for low-resource languages"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The entire zero-shot result rests on the assumption that off-the-shelf multilingual encoders place a query and its same-intent paraphrase in another language closer together than queries with different intents; if that cross-lingual alignment is weak for a language-model pair, nearest-neighbor labels carry no signal, as the XLM-R Swahili run (accuracy 0.119) shows.","fun_headline_variants_meta":{"raw":{"variants":["Zero-shot intents via similarity search on low-resource queries","Similarity search labels low-resource intents with no training data","No training needed: classify low-resource intents via nearest neighbors","Query similarity outsources intent labels for low-resource languages","Nearest-neighbor intent detection works for low-resource languages"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000605,"raw_usage":{"total_tokens":2753,"prompt_tokens":811,"completion_tokens":1942,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":427,"completion_tokens_details":{"reasoning_tokens":1855}},"tokens_in":427,"tokens_out":1942,"duration_ms":11883,"temperature":1.0,"reasoning_tokens":1855,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T14:36:02.463433+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Measure, on a large multilingual intent dataset, the probability that a query's nearest neighbor in a high-resource-language-only index has the same intent; if for a given language-encoder pair this probability is at chance level, the proposed zero-shot transfer is falsified for that pair, and this can be checked directly from the MASSIVE data used in the paper.","supporting_citations":[{"cited_title":"Nearest Neighbour Few-Shot Learning for Cross-lingual Classification","cited_arxiv_id":"2109.02221","evidence_quote":"This prior nearest-neighbor cross-lingual work is the direct comparison point; the paper's method differs by not fine-tuning and requiring zero target-language examples."},{"cited_title":"Hemphill, John J","cited_arxiv_id":null,"evidence_quote":"The ATIS dataset is used to grid-search the optimal $k$ for majority-vote label resolution."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"NLLB-200 supplies the machine-translation baseline that the proposed approach aims to be independent of."}],"review_version":1}