{"id":"e056045d-5b88-44fa-a1e1-7e980948ca37","arxiv_id":"2509.04011","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"Mid-layer LLM value vectors, projected through a contrastively trained MLP, enable zero-shot retrieval of documents by ad-hoc entity type.","lead":"This paper introduces NER Retriever, a system that retrieves documents mentioning entities of a user-defined type (for example, dinosaur) without a fixed label schema. It embeds entity mentions and type descriptions in a shared space using an intermediate layer of a frozen LLM plus a small contrastively trained projection network.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Few-NERD results are partly selected on the test set (layer/hyperparameter choice and sentence filter), so the headline 'substantially outperforms' is not yet established.","rationale":"The reader's weakest assumption was the entity span detector, and that is a real limitation: coverage of 0.89–0.94 costs about 11% R-Precision. However, the oracle-span experiments in Table 2 show the embedding itself can perform well when spans are known, so the detector issue does not directly threaten the core representation claim. The more load-bearing problem is that the Few-NERD result—the strongest evidence for the representation claim—is obtained under a protocol that selects the layer and hyperparameters using Few-NERD itself and modifies the Few-NERD test set after observing baseline behavior. This is not an accusation of fraud; it is a standard concern about selection on the test set. The paper is otherwise solid: it releases code, reports oracle experiments, and is transparent about the NERetrieve null result. If the proposed held-out test confirms the Few-NERD advantage, the paper's central claim would be strengthened; if not, the claim would need to be softened to a domain-specific or conditional finding. Since these are addressable evaluation issues rather than identified fatal flaws, the reader's CONDITIONAL verdict remains appropriate, with the condition being a pre-registered evaluation protocol.","tokens_in":11767,"tokens_out":7378,"duration_ms":74203,"concrete_test":"Re-run the Table 2 Few-NERD experiment under a held-out protocol: (a) do not drop sentences with fewer than five words; (b) choose the transformer layer/subcomponent and MLP hyperparameters on a 20% development split of Few-NERD (or on MultiCoNER 2/NERetrieve only), then evaluate on the untouched test split; (c) report NER Retriever's R-Precision and Wilcoxon p-value against E5-Mistral and NV-Embed v2 on that untouched test split. If the advantage over the best baseline is no longer significant, the paper's central claim is not robust to selection.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central evidence for the embedding claim is Table 2, especially the Few-NERD result (0.34 vs 0.04/0.08). That result is not an independent test of the method: Section 4 selects the representation (block 17, V-vector) by maximizing type-discrimination AUC on Few-NERD; Section 5.2 states that all hyperparameters were chosen by evaluating on the same discrimination task described in Section 4, which uses Few-NERD; and Section 6.3 applies a post-hoc filter to Few-NERD, omitting sentences shorter than five words after observing that they hurt sentence-encoder baselines. The NERetrieve result, where NER Retriever is not significantly better than NV-Embed v2 (0.28 vs 0.29), shows the advantage is not universal. If the Few-NERD advantage shrinks or loses significance when the layer/hyperparameters are chosen on a disjoint development set and the sentence filter is removed, then the strongest quantitative support for the central claim collapses. The oracle-span results (0.37) are useful, but they still use the same selected layer/hyperparameters and do not validate the automatic end-to-end claim independently.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes NER Retriever, a zero-shot named-entity retrieval framework. Instead of embedding entire sentences or documents, it indexes individual entity spans using the value vectors from block 17 of a frozen LLaMA 3.1 8B model, followed by a lightweight MLP trained with a triplet contrastive loss. At query time, a natural-language type description is embedded through the same pipeline and matched against the indexed entity vectors by cosine similarity. The method is evaluated on Few-NERD, MultiCoNER 2, and NERetrieve, reporting large gains over BM25 and sentence-level dense retrievers on the first two benchmarks and comparable performance on NERetrieve. The paper also includes a layer/component sweep across several LLMs, ablations, and an oracle-span analysis.","tokens_in":12070,"tokens_out":5005,"duration_ms":48224,"significance":"If the central claims hold, the paper makes a useful contribution: it provides evidence that mid-layer value vectors contain more type-discriminative information than top-layer outputs, and it shows that a compact projection of those vectors supports efficient, schema-free retrieval. The manuscript is generally well organized, includes a public codebase, and reports several useful ablations, including layer choice, token selection, and MLP projection. The oracle-span comparison is a good practice. However, the main quantitative evidence for the 'substantially outperforms' claim is weakened by selection on the test benchmark and by a post-hoc data filter, and the abstract overstates the NERetrieve result. These issues are fixable but require re-evaluation.","major_comments":[{"comment":"The Few-NERD evaluation is not an independent test of the method. Section 4 selects the representation source (block 17, V vectors) by maximizing type-discrimination AUC on Few-NERD, and Section 5.2 states that all numerical hyperparameters were selected by evaluating the same type-discrimination task described in Section 4, which uses Few-NERD. Few-NERD is then a test set in Table 2. This creates a selection effect: the reported 0.34 R-Precision and the 'substantially outperforms all baselines' claim for Few-NERD are not externally validated. Please re-run the layer/component selection and hyperparameter tuning on a disjoint development set (e.g., held-out Few-NERD types or another dataset) and re-report Table 2 with the test set used only once. The ablation in §8.1 inherits the same issue.","section":"§4, §5.2, Table 2"},{"comment":"The Few-NERD sentence-length exclusion is post-hoc. The paper states that instances with fewer than five words are omitted because 'such very short sentences were found to harm the performance of sentence-encoder baselines while not impacting our method.' This is an evaluation-set modification made after observing baseline behavior, and it is not applied to MultiCoNER 2 or NERetrieve. The reported Few-NERD score may depend on this filter. Please report results on the full Few-NERD set and on a pre-specified filtered version, and justify the filter on data-quality grounds rather than baseline performance. Also state whether the significance claim survives on the unfiltered set.","section":"§6.3, Few-NERD"},{"comment":"The abstract's blanket claim that 'NER Retriever significantly outperforms both lexical and dense sentence-level retrieval baselines' is contradicted by Table 2 and §7. On NERetrieve, NER Retriever achieves 0.28 R-Precision versus 0.29 for NV-Embed v2, and the paper itself says the difference is not statistically significant. The significance markers in Table 2 apply only to Few-NERD and MultiCoNER 2. Please revise the abstract and any summary statements to distinguish 'substantially outperforms on Few-NERD and MultiCoNER 2' from 'comparable on NERetrieve', or provide evidence for a stronger claim.","section":"Abstract and §7"},{"comment":"The end-to-end claim depends heavily on the automatic entity-span detector, but that dependency is only partially characterized. Appendix A.1 reports coverage of 0.89–0.94 across datasets, and §7 reports that oracle spans improve average R-Precision by roughly 11% (0.34 to 0.37 on Few-NERD, 0.32 to 0.35 on MultiCoNER 2, 0.28 to 0.34 on NERetrieve). Missed spans place an upper bound on recall that is independent of embedding quality. Please provide a sensitivity analysis over detector quality and state explicitly in the Limitations section that the reported automatic-system numbers are contingent on the extractor, not solely on the retrieval method.","section":"§6.1, §7, Appendix A.1"}],"minor_comments":[{"comment":"The caption contains a typo: 'entity focued' should be 'entity-focused'. The example text also alternates between 'Dinosaur' and 'dinosaurs'; standardize the query type.","section":"Figure 1"},{"comment":"The baseline is called 'NV-Embed v2' in most of the paper but 'Nvidia NV-Embed' in Table 3. Use one name consistently. Also, §6.2 says E5-Mistral and NV-Embed v2 'have comparable architecture and parameter count to LLaMA 3.1'; E5-Mistral is based on Mistral, not LLaMA, so clarify the comparison.","section":"Table 3 and §6.2"},{"comment":"The layer sweep uses 20 sampled fine-grained types and 20 sentences each. Please specify whether the sample comes from the Few-NERD train or test split, and report the exact type list or sampling seed; otherwise the selection procedure is not reproducible.","section":"§4"},{"comment":"The paper says the input layer size is 'derived from the LLM output (in our case 1024)' but LLaMA 3.1 8B has hidden size 4096. Clarify that 1024 is the value-vector dimension for the selected block, not the full hidden size.","section":"§5.2"},{"comment":"The text reports the EOS-based result as 0.03 R-Precision, while Table 4 reports 0.02 for '17V + EOS (-No MLP)'. Please correct the inconsistency.","section":"§8.2 and Table 4"}],"recommendation":"major_revision","confidential_remarks":"The central idea is plausible and the code release is a strength, but the Few-NERD result is currently not a clean evaluation because the same dataset is used for layer/hyperparameter selection and final testing. The sentence-length filter is also post-hoc. These are methodological fixes rather than fatal flaws, so I recommend major revision rather than rejection. The NERetrieve result should be reported accurately in the abstract, and the entity-detector dependence should be made a first-class limitation."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know two things about this paper. First, the central idea is genuinely new and useful: mid-layer value vectors from LLaMA 3.1 encode entity type distinctions far better than top-layer embeddings, and a light contrastive projection turns that into a workable zero-shot retrieval space. Second, the headline Few-NERD number (0.34 vs 0.04/0.08) is not as clean as it looks, because the layer, the hyperparameters, and even the sentence filter were all chosen on that same benchmark.\n\nWhat the paper does well: the layer sweep in Section 4 is thorough—416 representation sources, multiple LLMs, repeated with different seeds—and the finding that mid-layer V-vectors beat top-layer outputs is consistent across models. The ablation showing span-token representations beat EOS is clean. The oracle-span numbers (0.37/0.35/0.34) are the most convincing evidence that the embedding space is doing the work, and the paper is upfront about the entity-detector gap. Code is public.\n\nThe soft spots are real but addressable. The selection issue: Section 4 picks block 17/V-vector by maximizing AUC on Few-NERD; Section 5.2 says all hyperparameters were chosen on the same discrimination task; then Table 2 reports Few-NERD as a test set. That's a classic selection-on-test effect. A proper development split or nested evaluation would make the result trustworthy. The sentence filter in Section 6.3 is also a red flag: excluding sentences shorter than five words after seeing they hurt baselines is post-hoc, and if it doesn't affect the proposed method, that needs a mechanistic explanation. On NERetrieve, the method ties with NV-Embed v2 (0.28 vs 0.29, not significant), and the abstract's 'significantly outperforms' is simply wrong for that dataset; the body is honest about it, so the abstract should be corrected. Finally, the entity detector covers 0.89–0.94 of gold spans, and the oracle gap is around 11% R-Precision; the end-to-end claim depends on a third-party detector that isn't evaluated beyond coverage.\n\nWho is this for? People working on entity-centric retrieval, fine-grained typing, and representation selection inside LLMs. The method is practical and the negative NERetrieve result is informative. It deserves a serious referee, but not a quick accept: fix the selection issue, justify or remove the filter, align the abstract with the numbers.","headline":"Genuinely useful idea—mid-layer value vectors plus contrastive projection for entity retrieval—but the Few-NERD headline is partly selected on the test set, and the abstract overstates the NERetrieve result.","tokens_in":12548,"tokens_out":2222,"would_cite":true,"duration_ms":20204,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper argues that value vectors from a mid-layer transformer block, contrastively distilled, let any natural-language type description retrieve the matching entity mentions in zero-shot.","keywords":["named entity retrieval","zero-shot entity typing","type-aware embeddings","mid-layer representations","contrastive learning","dense retrieval","LLM internal states","entity span detection"],"falsifier":"Take a corpus with gold entity spans and a query set of unseen types. Run NER Retriever with oracle spans using (a) block-17 value vectors and (b) final-layer embeddings, both through the same trained projection. If (b) matches or beats (a), the central layer-selection claim fails. Alternatively, lower automatic span coverage below about 0.8 and show that end-to-end R-Precision collapses to baseline levels, which would confirm the detector bottleneck.","tokens_in":11659,"feed_emoji":"🔎","tokens_out":7223,"duration_ms":61932,"temperature":0.7,"pith_summary":"This paper takes on ad-hoc named entity retrieval: finding every document that mentions an entity of a user-defined type, where the type is given as a free-text description such as \"dinosaur\" or \"airline\" rather than a fixed schema. The authors try to show that the internal value vectors of a mid-layer transformer block—specifically block 17 of LLaMA 3.1 8B—carry entity-type information that final layers dilute, and that a lightweight contrastive projection can turn those vectors into a shared space where type descriptions and entity mentions sit close together. If the approach is right, it would make schema-free, zero-shot entity retrieval practical: index once with per-entity embeddings, then answer arbitrary type queries by nearest-neighbor search, without fine-tuning the LLM. Experiments on three benchmarks report that the retrieval stage outperforms lexical and sentence-level dense baselines on Few-NERD and MultiCoNER 2, and matches them on NERetrieve, with the whole system bottlenecked mainly by the upfront span detector.","feed_headline":"Mid-layer LLM vectors retrieve unseen entity types without fine-tuning","feed_subtitle":"Value vectors from block 17, contrastively compressed, beat lexical and dense baselines on two benchmarks.","key_machinery":"The value (V) vectors from transformer block 17 of LLaMA 3.1 8B—the self-attention value-projection output at the final token of each entity span—passed through a two-layer SiLU MLP trained with triplet contrastive loss into a 500-dimensional space, then searched by cosine nearest neighbor. This combination supplies the shared embedding space that aligns arbitrary type descriptions with the corresponding entity mentions.","core_discovery":"The central discovery is that representation choice inside a frozen LLM matters more than the usual final-layer default. Across 416 representation sources in LLaMA 3.1 8B, the value (V) vectors from self-attention block 17 give the best same-type-versus-different-type separation (AUC 0.78); a similar mid-layer peak appears in other tested architectures after normalizing depth. Feeding those vectors through a two-layer MLP trained with triplet loss produces a 500-dimensional type-aware embedding space in which entity mentions of the same type cluster, related fine-grained types sit nearby, and a user's type description retrieves entities by cosine similarity. With the full pipeline, R-Precisi","pith_inferences":["An implicit consequence: since type sensitivity peaks in mid-layers across several architectures, representation selection could improve other embedding-based tasks that currently default to top-layer pooling—entity linking, fine-grained typing, and open-domain question answering.","The training recipe uses only weakly labeled paragraphs from NERetrieve, so extending the same contrastive projection to specialized domains (law, medicine, finance) is a direct path to closing the limitation the authors state.","Because the index stores one vector per entity mention, it can be reused as evidence for retrieval-augmented agents: a broad query like \"texts mentioning a dinosaur\" can be decomposed into type-vector searches, which the paper gestures at but does not implement.","The 11-point oracle-span gap suggests a concrete engineering target: any better category-agnostic span detector should translate almost one-for-one into retrieval gains, offering a cheaper alternative to improving embeddings."],"forward_implications":["On Few-NERD and MultiCoNER 2, the full system beats BM25 and strong dense sentence embedders by roughly three to four times in R-Precision (0.34 vs 0.08; 0.32 vs 0.09).","With gold entity spans, the retrieval stage outperforms every baseline on all three benchmarks, so retrieval quality itself is not the current ceiling.","Choosing span-final-token representations over EOS or sentence tokens is decisive: 0.19 vs 0.03 in the no-MLP ablation on Few-NERD.","The contrastive projection contributes about half the retrieval score (0.16 without MLP vs 0.34 with it), and compresses vectors to 500 dimensions, cutting the index from 9.2 GB to 2 GB on MultiCoNER 2.","A zero-shot user never needs to predefine types: any natural-language type description is mapped through the same frozen LLM and projection, so novel and long-tail types can be queried directly."],"supporting_citations":[{"why":"Defines Named Entity Retrieval, supplies the NERetrieve benchmark and the weakly labeled training data used for the contrastive projection.","marker":"Katz et al., 2023"},{"why":"Supplies the Few-NERD benchmark and its 66 fine-grained entity types, used both for the layer-selection sweep and retrieval evaluation.","marker":"Ding et al., 2021"},{"why":"Supplies the MultiCoNER 2 benchmark, the short low-context English sentences used as a challenging retrieval test.","marker":"Fetahu et al., 2023"},{"why":"Supplies the CascadeNER entity-span extractor used in the detection stage; its measured coverage sets the ceiling on end-to-end performance.","marker":"Awasthy et al., 2020"},{"why":"Provides the triplet loss objective used to train the type-aware projection MLP.","marker":"Schroff et al., 2015"},{"why":"Motivates the BM25-based hard negative mining used in the contrastive training batches.","marker":"Karpukhin et al., 2020"},{"why":"Supplies E5-Mistral, the strong sentence-level dense retrieval baseline.","marker":"Wang et al., 2023"},{"why":"Supplies NV-Embed v2, the latent-attention sentence embedding baseline and the storage comparison point for index size.","marker":"Lee et al., 2024"},{"why":"Supplies BM25, the lexical retrieval baseline the paper must beat.","marker":"Robertson and Zaragoza, 2009"}],"fun_headline_variants":["Mid-layer LLM vectors beat baselines for unseen entity types","Block 17 value vectors enable zero-shot entity retrieval","Schema-free NER retrieval: mid-layer beats top-layer embeddings","Type-aware vectors from LLM mid-layer outscore lexical baselines","Zero-shot entity retrieval with mid-layer LLM representations"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The system's retrieval completeness rests on a category-agnostic entity span detector finding essentially all mentions; its measured coverage is 89–94%, and the missing spans cost about 11 percentage points of average R-Precision when replaced by gold spans.","fun_headline_variants_meta":{"raw":{"variants":["Mid-layer LLM vectors beat baselines for unseen entity types","Block 17 value vectors enable zero-shot entity retrieval","Schema-free NER retrieval: mid-layer beats top-layer embeddings","Type-aware vectors from LLM mid-layer outscore lexical baselines","Zero-shot entity retrieval with mid-layer LLM representations"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000185,"raw_usage":{"total_tokens":1168,"prompt_tokens":762,"completion_tokens":406,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":506,"completion_tokens_details":{"reasoning_tokens":337}},"tokens_in":506,"tokens_out":406,"duration_ms":4412,"temperature":1.0,"reasoning_tokens":337,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T10:26:46.265944+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a corpus with gold entity spans and a query set of unseen types. Run NER Retriever with oracle spans using (a) block-17 value vectors and (b) final-layer embeddings, both through the same trained projection. If (b) matches or beats (a), the central layer-selection claim fails. Alternatively, lower automatic span coverage below about 0.8 and show that end-to-end R-Precision collapses to baseline levels, which would confirm the detector bottleneck.","supporting_citations":[],"review_version":1}