{"id":"fdcc2ce6-9a21-4843-a038-ff76f24e5b55","arxiv_id":"2412.06654","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"An LLM-generated candidate list, averaged in embedding space and ranked against dictionary terms, outperforms several supervised reverse dictionary models on generalization splits.","lead":"GEAR is an unsupervised reverse dictionary method that asks an LLM to suggest candidate words for a definition, embeds those candidates, averages the embeddings, and searches a dictionary. The paper reports that this simple pipeline beats several supervised baselines on two of three Hill dataset test splits, but the comparison is weakened by possible LLM training-data contamination and missing recent baselines.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"LLM memorization of Hill test definitions may drive the SOTA gains; no contamination control is provided, so the 'new state of the art' claim is not yet established.","rationale":"I agree with the reader's weakest_assumption: the 'unseen' and 'description' splits are the load-bearing evidence for the SOTA claim, and memorization is the most direct threat to that evidence. The GEAR pipeline's generate step is a free-form LLM call; GPT-4o mini has been trained on essentially all public English text, including the dictionaries that constitute Hill's test set. The paper even cites WordNet and Wiktionary as sources. So the high acc@1/acc@100 numbers on 'unseen' (Table 2) may reflect retrieval from pretraining, not generalization. This is not an accusation; it is a standard evaluation hazard. The correct response is a control experiment. My proposed paraphrase test is a feasible way to hold the generator's reasoning constant while removing verbatim overlap; if the advantage persists, the claim is strengthened; if not, the headline must be softened. Other issues (missing recent baselines, no significance testing, Table 4's implausible rp P@5 of 75.89) are secondary; they affect the strength of the comparison but not the core logic. Given that the method is simple, reproducible (code provided), and component-analyzed, the appropriate verdict is conditional acceptance contingent on the contamination control. This matches the reader's verdict, so I recommend no change.","tokens_in":14227,"tokens_out":6708,"duration_ms":69622,"concrete_test":"Two-part check: (1) Paraphrase every definition in Hill's unseen and description splits with an independent LLM (e.g., GPT-4 or a different instruction-tuned model) under instructions to preserve meaning but change surface form; confirm via n-gram overlap that the paraphrases share <20% 4-grams with the originals. (2) Rerun the exact GEAR pipeline (bp2 prompt, Instructor-large embeddings, same KNN vocabulary) on these paraphrased definitions. Compare acc@1/acc@100 against Table 2. If acc@1 on the unseen split drops by more than 15 absolute points, or falls below the best supervised baseline (BERT/RoBERTa from Yan et al. 2020), the SOTA result is likely driven by memorization. If performance is roughly stable, the reasoning-based explanation survives.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—GEAR sets a new state of the art on Hill's 'unseen' and 'description' splits (Section 5.1, Table 2)—rests on the assumption that GPT-4o mini's generated candidates reflect reasoning from the definition rather than memorization of the test pairs during pretraining. This assumption is unexamined and likely false. Hill's 'unseen' definitions are sourced from WordNet, Webster's, and Wiktionary; these exact definition-word pairings are almost certainly present in GPT-4o mini's web-scale training data. The 'unseen' split is thus unseen only for the supervised baselines, not for the LLM generator. If the LLM can surface the gold word as a candidate, the subsequent embedding-and-KNN steps will rank it highly, making the comparison to baselines (trained solely on Hill's train split) an apples-to-oranges claim. The paper offers no contamination test, no paraphrase control, no membership inference, and no generator with transparent training data (the Llama 3.1-70B replication in Section 7 does not help, as its data is also web-scale). The Limitations section omits this threat entirely, despite acknowledging other, less central limitations. Without a control for memorization, the 'new SOTA' claim is not established.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces GEAR, an unsupervised reverse-dictionary (RD) pipeline that (1) prompts an LLM (GPT-4o mini) to generate candidate terms for a definition, (2) embeds those candidates with a text encoder, (3) averages the embeddings, and (4) performs KNN search over the target dictionary's vocabulary. The authors evaluate on Hill et al.'s (2016) three test splits and on a multi-dictionary resource (3D-EX) with several embedding models and three prompt variants. They claim GEAR outperforms supervised baselines and sets a new state of the art on Hill's 'unseen' and 'description' splits, while also analyzing how dictionary register and domain affect performance.","tokens_in":14403,"tokens_out":8766,"duration_ms":84594,"significance":"If the central claim holds, GEAR would be a notably simple and practical unsupervised alternative to supervised RD systems, with the added advantage of not requiring training on a specific dictionary. The paper includes reproducible code and a comparative analysis across diverse dictionary styles, which is valuable for the RD community. However, the headline result rests on the assumption that the LLM generates candidates from understanding the definition rather than from memorizing the test definitions during pretraining; this assumption is untested and is the main risk to the paper's contribution.","major_comments":[{"comment":"The 'unseen' and 'description' splits of Hill et al. are drawn from WordNet, Webster's, and Wiktionary, which are very likely present in the pretraining data of GPT-4o mini and Llama 3.1-70B. As a result, the model may surface the gold word from memory, and the 'unseen' condition is not unseen for the LLM. The lack of any contamination control (e.g., paraphrased definitions, membership inference, or post-cutoff vocabulary tests) means the reported 'new state of the art' is not established. Please add such controls or explicitly re-frame the results as measuring both memory and generalization.","section":"Section 5.1 (Table 2), Section 7 (Table 6), Limitations"},{"comment":"The abstract and Section 1 state that GEAR outperforms 'supervised baselines (including those leveraging LLMs)', but Table 2 only includes BiLSTM/transformer-encoder baselines from Zhang et al. (2020) and Yan et al. (2020); the only LLM-based supervised method discussed (Tian et al., 2024) is not compared. The claim therefore overstates the evidence. Please either include an LLM-based supervised baseline or temper the wording.","section":"Abstract and Section 1"},{"comment":"The number of generated candidates k and the decoding settings (temperature, sampling strategy) are not reported for the main experiments, even though Figure 2 shows that k affects performance. This omission makes the results difficult to reproduce exactly; the authors should state the default k and generation hyperparameters.","section":"Section 6.1 and Figure 2"}],"minor_comments":[{"comment":"In the seen split, rp row, the value '4.6' for ACC@100 appears to be a typo for '94.6'.","section":"Table 7"},{"comment":"The instructor instruction variants (e.g., 'dict. - dict.') are not defined in the caption; please clarify what each pair refers to.","section":"Table 5"},{"comment":"There is a typo 'SoTa' and inconsistent capitalization of '3D-Ex' vs. '3D-EX' throughout the paper.","section":"Section 5.2"},{"comment":"The Limitations section acknowledges the absence of baseline comparisons on 3D-EX but does not mention the potential pretraining contamination of the LLM, which is a more consequential threat to the paper's main claim.","section":"Limitations"},{"comment":"The paper does not report standard deviations or confidence intervals for the LLM-based results; since LLM generation is stochastic unless greedy decoding is used, a small number of repeated runs would strengthen the comparison.","section":"General evaluation"}],"recommendation":"major_revision","confidential_remarks":"The paper has a clear risk of benchmark contamination that the authors do not acknowledge; this should be taken seriously during revision. The authors' framing of 'new state of the art' should be softened until they demonstrate robustness to paraphrasing. Also, the comparison to supervised baselines is somewhat dated; adding a recent LLM-based RD system would make the claim more convincing."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Read it. The GEAR pipeline is a clean idea: generate candidate terms with an LLM, embed each, average the embeddings, and KNN-search the dictionary vocabulary. As far as the cited work goes, that specific assembly is new for reverse dictionary, and it comes with open code and a reasonably thought-out evaluation across eight dictionaries. The 3D-EX analysis, showing which dictionary registers are easy and hard, is a genuinely useful resource.\n\nThe problem is the headline claim. \"Outperforms supervised baselines\" and \"new state of the art\" on Hill's unseen and description splits is not established. The LLM generators (GPT-4o mini and Llama 3.1-70B) are trained on web-scale data that almost certainly contains the very WordNet and Wiktionary definitions used in those test splits. So \"unseen\" is unseen only for the supervised baselines, not for the generator. The paper provides no contamination test, no paraphrase control, no membership inference, and the Limitations section never mentions this threat. That is a load-bearing omission, not a minor one. If the model has memorized the test pairs, GEAR's numbers reflect retrieval of memorized strings, not generalization from definitions.\n\nThere are also smaller but real issues: the SOTA comparison omits recent supervised systems cited in the paper's own related work (e.g., Tian et al. 2024), and no error bars or significance testing are reported. The tables contain obvious typos—Table 4's P@5 of 75.89 for rp on 3D-EX and Table 7's ACC@100 of 4.6 on the seen split are both impossible as printed. Typos happen, but two in the main results tables make you double-check every number.\n\nNone of this kills the underlying method. Even as a purely unsupervised baseline, GEAR is simple, scalable, and likely useful for practitioners who need a fast RD system without training. The paper deserves peer review, but the authors need to do real work: measure contamination (e.g., paraphrase the definitions, test on definitions written after the model's cutoff, or use a model with documented training data), add the missing baselines, report variance, and fix the typos. If contamination cannot be ruled out, the SOTA claim should be softened or removed.\n\nWho is this for? People working on lexical access, semantic lexicons, and anyone designing LLM benchmarks—the memorization question is a warning for the whole field. Bring it to reading group; the discussion will be good.","headline":"A clean unsupervised RD pipeline with a serious unaddressed contamination problem: the SOTA claim rests on test data the LLM generator has likely memorized.","tokens_in":15004,"tokens_out":3213,"would_cite":true,"duration_ms":30432,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"GEAR, a simple unsupervised generate-embed-average-rank pipeline, outperforms supervised reverse dictionary systems on the generalization splits of Hill's benchmark.","keywords":["reverse dictionary","unsupervised method","large language models","text embeddings","generate-embed-average-rank","nearest neighbor search","dictionary evaluation","lexical access"],"falsifier":"Build a reverse-dictionary test set from terms coined after GPT-4o mini's knowledge cutoff (or from an invented lexicon with definitions written in Hill's style) and evaluate GEAR on it; if accuracy falls to the level of the supervised baselines, the claimed generalization is likely memorization, while sustained high accuracy would confirm the reasoning-based account.","tokens_in":13947,"feed_emoji":"🔍","tokens_out":11978,"duration_ms":103080,"temperature":0.7,"pith_summary":"GEAR is a four-step, fully unsupervised recipe for reverse dictionary lookup: given a definition, an LLM generates candidate terms, an embedding model turns each candidate into a vector, the vectors are averaged into a centroid, and the centroid is matched against the dictionary's vocabulary by nearest-neighbour search. The paper argues that this zero-training pipeline outperforms heavily tuned supervised baselines, including LLM-based ones, on the two Hill et al. test splits that demand generalization (unseen definitions and human-written descriptions), while overfitting less. The broader point the authors want to establish is that the lexical knowledge needed to retrieve the right word from a definition already lives inside pretrained LLMs and can be harvested without any gradient updates, just by averaging a few generated candidates' embeddings. If right, reverse dictionary systems become dramatically cheaper to build and to port across dictionaries of widely different registers, from WordNet to Urban Dictionary.","feed_headline":"Unsupervised GEAR pipeline beats supervised reverse dictionaries","feed_subtitle":"A generate-embed-average-rank recipe sets new state of the art on Hill's generalization splits.","key_machinery":"The central machinery is the GEAR pipeline itself, which consists of four named steps: Generate, Embed, Average, Rank. Given a definition, an LLM produces a set of candidate terms; each candidate is embedded with a text encoder; the candidate vectors are mean-pooled into a single centroid; and that centroid is used for KNN search over the dictionary's full term vocabulary via cosine similarity. The averaging step is load-bearing: a single generated candidate is suboptimal, and pooling a few candidates (performance plateaus at 2-3) yields a stable centroid that lands closer to the target term's embedding than any individual guess, acting as a cheap ensemble over the LLM's stochasticity. The paper also finds that the embedding model's instruction schema matters (dictionary-specific instructions for Instructor help most), and that max-pooling consistently underperforms averaging.","core_discovery":"On the paper's own terms, the central discovery is that a fully unsupervised four-step pipeline, GEAR (generate, embed, average, rank), performs reverse dictionary lookup without any training. For each input definition, the LLM (GPT-4o mini, or the open-source Llama 3.1-70B) produces a ranked list of candidate terms; GEAR embeds each candidate with a sentence encoder (best results with Instructor), averages the candidate vectors into a single centroid, and returns the dictionary terms closest to that centroid by cosine similarity. On Hill's benchmark, GEAR sets a new state of the art on the unseen split (accuracy@1 of 70.0% with the example-augmented base prompt) and on the human-description split (accuracy@1 of 92.5%), surpassing supervised systems such as Multi-channel, BERT, and RoBERTa; it trails only on the seen split, where MS-LSTM's near-perfect score reflects memorization of the training data. Across the nine dictionaries in 3D-EX, GEAR consistently improves over LLM-only generation and over untuned embedding baselines, with the largest gains on specialized resources like Hei++ and Sci-definition and the smallest on slang (Urban) and CHA, showing that the method is dictionary-agnostic while register still matters.","pith_inferences":["If the contamination concern is set aside, GEAR's success suggests that the lexical knowledge required for reverse dictionary lookup is almost entirely present in pretrained LLMs and extractable without gradient updates; a testable corollary is that the same generate-embed-average recipe should transfer to other description-to-item retrieval tasks, such as code search from docstrings or product lo","The averaging step likely works by reducing the variance of the LLM's individual guesses rather than by adding lexical knowledge from the embeddings; this could be tested by comparing GEAR with an LLM-only reranker that scores the same candidate set.","Because GEAR is unsupervised and dictionary-agnostic, it could be applied to fresh or rapidly evolving lexicons (internal company terminology, emerging slang, rare technical registers) without collecting labeled pairs, with the remaining bottleneck being the LLM's candidate quality on niche registers.","The finding that untuned embeddings alone are far below the LLM-only baseline but competitive on technical dictionaries suggests that retrieval-based and generation-based RD should be treated as complementary signals rather than rivals."],"forward_implications":["Unsupervised reverse dictionary lookup can replace supervised training: on two of Hill's three test splits, the zero-training GEAR pipeline exceeds tuned supervised systems and overfits less.","The pipeline transfers across dictionaries without retraining: GEAR improves over LLM-only generation on all nine 3D-EX sources, with the largest gains on specialized technical resources such as Hei++ and Sci-definition.","Embeddings alone are not competitive, but they are essential inside GEAR: untuned embeddings score well below an LLM-only baseline on average, yet in combination with generated candidates they lift MRR from the 28-31 range to the 43-46 range across prompt variants.","Prompt engineering contributes little for generation alone but more once embeddings are added: the reasoning prompt adds roughly 7% MRR within GEAR versus about 2% for LLM-only generation.","Only a handful of generated candidates are needed: performance plateaus at 2-3 averaged terms, keeping the method's computational overhead small."],"supporting_citations":[{"why":"supplies the benchmark dataset with seen, unseen, and human-description splits, and the BOW/RNN baseline results that GEAR is compared against.","marker":"Hill et al. (2016)"},{"why":"provides the Multi-channel supervised baseline and its reported results on Hill's dataset, which GEAR outperforms on generalization splits.","marker":"Zhang et al. (2020)"},{"why":"provides the BERT and RoBERTa supervised baselines on Hill's dataset that GEAR beats on unseen and description splits.","marker":"Yan et al. (2020)"},{"why":"describes the Instructor embedding model, whose dictionary-specific instructions produce GEAR's best retrieval results.","marker":"Su et al. (2022)"},{"why":"describes SBERT, the alternative sentence encoder evaluated within GEAR's embed step.","marker":"Reimers and Gurevych (2019)"},{"why":"is the reference for GPT-4o mini, the LLM used in the generate step for most GEAR experiments.","marker":"Achiam et al. (2023)"},{"why":"introduces the 3D-EX multi-dictionary dataset used for the cross-register generalization experiments.","marker":"Almeman et al. (2023)"},{"why":"is the reference for the open-source Llama model used to show that GEAR does not depend on a proprietary LLM.","marker":"Touvron et al. (2023)"}],"fun_headline_variants":["GEAR: unsupervised reverse dictionary beats supervised on Hill's unseen split","Generate-embed-average-rank: simple unsupervised RD tops supervised Baselines","GEAR sets new SOTA on Hill's generalization splits without any training","Unsupervised GEAR uses LLM + embeddings to beat trained RD baselines"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The headline results stand on the assumption that GPT-4o mini's correct answers on the 'unseen' and 'description' splits come from reasoning about the definitions rather than from having memorized those exact word-definition pairs during pretraining; the paper reports no contamination test, and if the model has already seen Hill's test definitions, those splits are only 'unseen' for the supervised baselines.","fun_headline_variants_meta":{"raw":{"variants":["GEAR: unsupervised reverse dictionary beats supervised on Hill's unseen split","Generate-embed-average-rank: simple unsupervised RD tops supervised Baselines","GEAR sets new SOTA on Hill's generalization splits without any training","Unsupervised GEAR uses LLM + embeddings to beat trained RD baselines"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000673,"raw_usage":{"total_tokens":3079,"prompt_tokens":977,"completion_tokens":2102,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":593,"completion_tokens_details":{"reasoning_tokens":2023}},"tokens_in":593,"tokens_out":2102,"duration_ms":17328,"temperature":1.0,"reasoning_tokens":2023,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T19:25:58.607839+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Build a reverse-dictionary test set from terms coined after GPT-4o mini's knowledge cutoff (or from an invented lexicon with definitions written in Hill's style) and evaluate GEAR on it; if accuracy falls to the level of the supervised baselines, the claimed generalization is likely memorization, while sustained high accuracy would confirm the reasoning-based account.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"supplies the benchmark dataset with seen, unseen, and human-description splits, and the BOW/RNN baseline results that GEAR is compared against."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"provides the Multi-channel supervised baseline and its reported results on Hill's dataset, which GEAR outperforms on generalization splits."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"introduces the 3D-EX multi-dictionary dataset used for the cross-register generalization experiments."}],"review_version":1}