{"id":"14451184-905d-47de-aec7-407cf1c3b80b","arxiv_id":"2412.02228","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"BANER combines boundary-aware contrastive learning with LoRAHub composition to improve few-shot NER with LLaMA-2-7B, claiming state-of-the-art F1 scores on Few-NERD and four cross-dataset benchmarks.","lead":"This paper introduces BANER, a two-stage few-shot named entity recognition system that uses a large language model with a boundary-aware contrastive loss for span detection and LoRAHub domain composition for type classification. If the reported gains hold, it offers a practical recipe for adapting a 7-billion-parameter LLM to new entity types with only a few dozen labeled examples.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Hidden layer is tuned on the GUM 5-shot test set and then reported in the same table, so the key cross-dataset results are optimistically biased.","rationale":"The paper's central claim is an empirical one, so the validity of the headline rests on the fairness of the evaluation protocol. The hidden-layer selection in §4.5 is the clearest point where the protocol is not fair: the same GUM 5-shot configuration is used both to choose a training hyperparameter and to report the resulting score. This is a direct violation of the standard train/validation/test separation. The concern is not that the authors are dishonest; it is that the reported numbers are optimistically biased in an unknown amount. If a proper validation split produces a smaller or nonsignificant advantage over TadNER/BDCP, the claim that BANER 'outperforms prior methods' would still be true on some datasets but not uniformly, and the marginal value of the boundary-aware loss would need re-quantification. Other issues (the ambiguous LoRAHub composition in Eq. 14, notational problems in Eqs. 6–8, and the use of a 7B LLM against mostly BERT-scale baselines) are real but secondary: they affect reproducibility and interpretation, not the core validity of the empirical comparison as directly as the test-set tuning does. The paper has genuine strengths: a coherent two-stage design, ablations that mostly support the components, and a promised public code release, so the appropriate outcome is a conditional acceptance that requires a leakage-free re-evaluation.","tokens_in":16076,"tokens_out":10194,"duration_ms":101332,"concrete_test":"Re-run the cross-dataset evaluation with the hidden layer selected on a held-out validation split disjoint from the Table 4 test sets (e.g., half of the GUM support set or a source-domain validation set), then report GUM 5-shot and the cross-dataset average. If GUM 5-shot F1 drops by more than 1 point, or if the average margin over the strongest baseline (TadNER or BDCP) narrows below the reported 5.1-point 5-shot advantage, the headline should be revised to acknowledge the test-set-driven hyperparameter choice.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4.5 selects the hidden layer for the boundary-aware contrastive loss by comparing performance 'under the GUM 5-shot setting.' GUM 5-shot is then reported in Table 4 as one of the four target-domain test sets. This is test-set-driven hyperparameter selection: the layer that maximizes performance on GUM 5-shot is used to produce the GUM 5-shot entry and all other cross-dataset entries. The reported GUM 5-shot F1 (44.0) and the cross-dataset average (53.9) therefore cannot be read as unbiased estimates of BANER's effectiveness. Because the same layer choice also affects the ablation study (Table 5), the contribution of boundary-aware contrastive learning may be overstated. The central claim 'outperforms prior methods, validating its effectiveness' depends on exactly these numbers, so the evaluation protocol must be fixed before the claim is fully accepted. A fair protocol would select the hidden layer on a validation split disjoint from the reported test set (e.g., a split of the source-domain data or a held-out development set within GUM), and then report results on the untouched test sets.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes BANER, a two-stage few-shot NER framework built on LLaMA-2-7B with LoRA. The first stage performs span detection with an auto-regressive generation loss augmented by a boundary-aware contrastive loss that pulls entity-type phrase embeddings toward the embeddings of true span boundary tokens. The second stage performs type classification via prototypical networks, with per-domain LoRA modules composed by LoRAHub for cross-domain adaptation. The authors report state-of-the-art F1 scores on Few-NERD and four cross-dataset benchmarks, and provide ablations and experiments with Mistral-7B and LLaMA-3-8B.","tokens_in":16347,"tokens_out":6635,"duration_ms":62622,"significance":"If the reported results are unbiased, the paper demonstrates a practical recipe for making 7B-scale LLMs effective few-shot NER systems, with contributions in boundary-aware contrastive learning and LoRA-based domain adaptation. The release of code and data is a concrete strength, as are the ablations that isolate individual components and the evaluation across multiple LLM architectures. However, the significance is currently limited by evaluation-protocol issues: test-set-based selection of the contrastive layer, an underspecified LoRAHub weight-optimization protocol, and a main comparison that does not control for base-model scale. These concerns must be resolved before the state-of-the-art claim can be fully trusted.","major_comments":[{"comment":"The hidden layer index (25th) for the boundary-aware contrastive loss is selected by comparing performance on the GUM 5-shot setting, and GUM 5-shot is itself one of the reported test sets in Table 4 and is included in the cross-dataset average. This is test-set-driven hyperparameter selection: the reported GUM 5-shot F1 and the cross-dataset averages are optimistically biased, and the same layer choice propagates into the ablation study in Table 5. Please re-select the layer on a validation split disjoint from all test sets (e.g., a held-out portion of the source-domain data or a GUM development split) and re-report the main tables and ablations under that protocol.","section":"Section 4.5, Table 4, Figure 4"},{"comment":"The LoRAHub composition protocol is not fully specified. The text does not state whether the weights w are optimized on the support set, on a separate validation set, or on the query (test) set; the L1 coefficient alpha is never assigned a value in Section 4.1.4; and Eq. (14) contains a typographical error ('w1A1 ... + wN AN' should presumably be a sum). If w is fitted on the query set, the cross-dataset results are transductive and not comparable to the published baselines. Please specify the exact optimization procedure, the data split used to determine w, the value of alpha, and the initialization of w.","section":"Section 3.2.3, Eq. (14), Section 4.1.4"},{"comment":"The boundary-aware contrastive loss is not written with consistent notation. In Eq. (6), L_cl is a sum over the batch index i, while Eqs. (7)-(8) define sim(o,pos_i) and sim(o,neg_i) as sums over m spans that return B-dimensional vectors. Moreover, e_o, e_posi, and e_negi have shapes B x 1 x D, B x 2 x D, and B x 4 x D, so the dot products in Eqs. (7)-(8) are not well-defined without a pooling or flattening operation. Please rewrite the loss with consistent indices and explicit tensor shapes (e.g., mean-pooling over the span dimensions) so that the method is reproducible.","section":"Section 3.1.2, Eqs. (6)-(8)"},{"comment":"The definition of the span representation z_i used to compute prototypes is missing. Eq. (10) averages z_i, but the paper never states how z_i is extracted from the LLM hidden states (which layer, whether the span is mean-pooled, whether the entity type phrase embedding is used). Without this, the type classification stage cannot be reproduced. Please provide the exact construction of z_i.","section":"Section 3.2.1, Eq. (10)"},{"comment":"The main comparison does not control for base-model scale: BANER is built on LLaMA-2-7B, while most baselines (ProtoBERT, NNShot, StructShot, ESD, DecomposedMetaNER, TadNER, TSFNER, BDCP) are based on substantially smaller pretrained encoders. The reported gains in Tables 3 and 4 may therefore reflect capacity rather than the proposed boundary-aware contrastive loss or LoRAHub composition. The authors should add same-backbone baselines (e.g., LLaMA-2-7B fine-tuned with LoRA in the same two-stage framework but without the proposed components) or restrict the claim to 'BANER outperforms prior methods' only under the caveat that the backbones differ.","section":"Tables 3-4"}],"minor_comments":[{"comment":"The text says the experiments are run 'under the GUM 5-shot setting' but then mentions both 1-shot and 5-shot scenarios; please clarify whether Figure 3 reports one or both settings.","section":"Section 4.4, Figure 3"},{"comment":"The quantity d(p,s) in Eq. (12) is a cosine similarity, not a distance; rename it to 'similarity' or use a true distance (e.g., 1 minus cosine similarity) to avoid ambiguity.","section":"Section 3.2.2, Eq. (12)"},{"comment":"The claim that this is 'the first integration of LLM with contrastive learning for few-shot NER tasks' should be checked against a broader set of recent works; if uncertain, soften the claim to 'to our knowledge' with a more clearly scoped comparison.","section":"Contributions, Introduction"},{"comment":"The caption contains an apparent typo: 'LoRAWiki DomainSocial Domain' appears to be a line-break error in the figure text; please fix the caption and the accompanying figure.","section":"Figure 2 caption"},{"comment":"The implementation details state that no validation set is used during source-domain training; given the test-set-based layer selection in Section 4.5, please add a validation split for hyperparameter selection as recommended in Major Comment 1 and describe how it is used.","section":"Section 4.1.4"}],"recommendation":"major_revision","confidential_remarks":"The two main blockers are the test-set-driven selection of the contrastive layer and the underspecified LoRAHub weight optimization. Both are fixable: re-running with a proper validation split and clearly specifying the LoRAHub protocol would address the first two major comments. The base-model-scale issue is more a matter of presentation and fair comparison; it may require additional baselines. I would not reject outright because the core idea is plausible, the ablations are informative, and the code release supports verification."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: BANER is a genuinely reasonable idea packaged with an evaluation that is partly compromised by test-set tuning. The paper deserves a serious referee, but the cross-dataset headline numbers cannot be taken at face value until the hidden-layer selection is moved off the GUM test set.\n\nThe new piece is the boundary-aware contrastive loss, where the entity type phrase in the prompt is used as an anchor and the concatenated first/last token embeddings of a true span are pulled toward it, while outside tokens are pushed away. That is a clean idea, and the two-stage decomposition is sensible. The ablations are internally consistent: removing the contrastive component drops F1 notably, and removing LoRAHub composition also hurts. The fact that the method transfers to Mistral-7B and LLaMA-3-8B is a plus. Code and data are promised.\n\nThe soft spots, in order of seriousness. First, Section 4.5 selects the hidden layer (the 25th) by comparing performance under the GUM 5-shot setting, and that same GUM 5-shot result is reported in Table 4. That is test-set-driven hyperparameter selection. The GUM entry and the cross-dataset average are therefore optimistically biased, and the ablation in Table 5 inherits that bias. This needs to be fixed before the headline claim is credible. Second, the LoRAHub composition protocol is under-specified. A LoRA module is fine-tuned per target domain, then weights w are fit by minimizing cross-entropy on 'the target domain'—it is not clear which split is used for this fitting, or whether the composition is re-fit per evaluation episode. If the same support set used to compute prototypes is also used to fit the composition weights, that is another leak. Third, the loss equations in Section 3.1.2 are sloppy: the batch index and span index are conflated, and the formula for Lcl does not match a standard contrastive objective. These are fixable but confusing. Fourth, the baselines are mostly BERT-scale models; the strongest control is the paper's own 'w/o BASD' ablation rather than a head-to-head with another LLM-based method.\n\nNone of this means the idea is wrong. The mechanism is plausible and the ablations are at least internally consistent. But the reported numbers should be treated as upper bounds until the evaluation protocol is tightened.\n\nWho should read this: anyone working on few-shot NER with generative LLMs will want to know about the contrastive span-boundary trick. It is worth a serious referee, with the eval protocol as the main issue to resolve.","headline":"A plausible boundary-aware contrastive recipe for few-shot NER with LLMs, but the headline numbers are partly compromised by hidden-layer selection on the GUM test set.","tokens_in":16834,"tokens_out":2830,"would_cite":false,"duration_ms":26762,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"BANER claims that a LLaMA-2-7B fine-tuned with boundary-aware contrastive learning and LoRAHub composition outperforms prior methods on few-shot NER benchmarks.","keywords":["few-shot named entity recognition","boundary-aware contrastive learning","large language models","LoRA","domain adaptation","prototypical networks","span detection","cross-domain NER"],"falsifier":"Take the span-detection stage with and without the boundary-aware contrastive loss under identical LoRA fine-tuning and evaluate on a held-out dataset such as WNUT 5-shot; if removing the loss does not reduce over- and under-detected spans or lower F1, the loss is not doing the claimed work. A second check is to repeat the hidden-layer comparison from Section 4.5 on a different dataset; if the 25th layer is not the best there, the layer choice was overfit to GUM.","tokens_in":15880,"feed_emoji":"🏷️","tokens_out":8820,"duration_ms":77182,"temperature":0.7,"pith_summary":"BANER is a two-stage approach to few-shot named entity recognition built on a large language model. The paper claims that adding a boundary-aware contrastive loss during the span-detection stage fixes the over- and under-detected spans that plague two-stage prototypical networks, while composing per-domain LoRA modules with LoRAHub aligns target-domain entity prototypes with source-domain knowledge during type classification. If correct, this means a 7-billion-parameter model can be adapted to new entity types with only a handful of labeled examples and outperform both one-stage and prior two-stage few-shot NER methods. The authors report gains over the strongest baselines on Few-NERD and on four cross-dataset benchmarks, and show the same recipe improves Mistral-7B and LLaMA-3-8B under the GUM 5-shot setting. That matters because general-purpose LLMs have not usually proven to be reliable few-shot information extractors, and few-shot NER is the setting where new entity vocabularies appear constantly.","feed_headline":"LLM with boundary-aware contrastive learning tops few-shot NER","feed_subtitle":"Two-stage span detection plus LoRA domain alignment lifts F1 over prior methods across five benchmarks.","key_machinery":"The load-bearing object is the boundary-aware contrastive loss $\\mathcal{L}_{cl}$ computed from hidden states of the 25th layer of LLaMA-2. For each candidate span, the positive sample is the concatenation $[h_{b_i}, h_{e_i-1}]$ of the hidden states of the span's first and last token, the negative sample is the concatenation of the hidden states of the two tokens before and two tokens after the span, and the anchor is the embedding of the entity-type phrase in the prompt; the loss is $-\\frac{1}{B}\\sum_{i=1}^{B}\\log\\sigma(\\text{sim}(o, pos_i) - \\text{sim}(o, neg_i))$ with cosine similarities. This loss is added to the autoregressive span-generation loss with weight $\\lambda = 0.001$. The second mechanism is LoRAHub composition $\\hat{m} = (w_1 A_1 + \\cdots + w_N A_N)(w_1 B_1 + \\cdots + w_N B_N)$, learned by minimizing cross-entropy plus L1 regularization on the weights, which merges per-domain LoRA adapters so target-domain prototypes stay aligned with source-domain knowledge.","core_discovery":"The paper's central claim is that few-shot NER improves when an LLM is taught entity boundaries directly and when type classification is adapted across domains in a parameter-efficient way. In the span-detection stage, BANER fine-tunes the LLM with the standard generation loss plus a boundary-aware contrastive loss: the embedding of the entity-type phrase is pulled toward the concatenated hidden states of the true span's first and last token and pushed away from the hidden states of tokens just outside the span. In the type-classification stage, a prototypical network is built on the LLM's span representations, separate LoRA modules are trained per target domain, and LoRAHub composes those modules to align target-domain prototypes with source-domain knowledge. Experiments on Few-NERD and on the OntoNotes-to-I2B2, CoNLL, WNUT, and GUM transfer settings report that BANER surpasses the compared one-stage and two-stage baselines, including average F1 gains of 5.2 points on Few-NERD intra and 2.3 and 5.1 points on the 1-shot and 5-shot cross-dataset settings.","pith_inferences":["Because the hidden layer for the contrastive embeddings was selected on the GUM 5-shot set, the strongest test of the mechanism would hold out the layer choice on a second dataset; if layer 25 is not consistently best, part of the reported margin may come from fitting that evaluation set.","The contrastive loss uses only the span's first and last token hidden states; a natural variant would include interior span tokens or full-span pooling, which could improve recall on long entities or numeric spans.","LoRAHub's benefit likely grows with the number of target domains available at composition time; in a pure single-domain 5-shot transfer to one dataset its advantage over one fine-tuned LoRA module may be smaller than the ablation suggests.","A direct probe of the mechanism would inspect the hidden-state geometry before and after training: if the loss works as intended, the cosine similarity between entity-type phrase embeddings and true-span boundary embeddings should increase relative to outside-token embeddings."],"forward_implications":["A single 7B LLM can be turned into a competitive few-shot NER system by training only low-rank adapters, not full model weights.","Boundary-aware contrastive learning reduces false spans early, so the type-classification stage receives cleaner candidates and the two-stage pipeline errors do not compound.","LoRAHub composition means a model pretrained once on a source domain can be adapted to several target domains by training and then reweighting small domain-specific modules.","The same two enhancements transfer to other open LLMs, so the recipe is not tied to one backbone.","The method's largest margins are reported on dense-entity and intra-domain settings where boundary errors dominate, suggesting boundary perception is the bottleneck being addressed."],"supporting_citations":[{"why":"It provides the Few-NERD dataset and the N-way K-shot episodic evaluation protocol used in the main results.","marker":"(Ding et al., 2021)"},{"why":"It supplies the CONTaiNER contrastive baseline and the cross-dataset few-shot evaluation protocol with sampled support sets.","marker":"(Das et al., 2022)"},{"why":"It introduces DecomposedMetaNER, the two-stage prototypical baseline whose framework BANER extends and whose episode evaluation is followed.","marker":"(Ma et al., 2022b)"},{"why":"It presents TadNER, the strongest two-stage baseline, with type-aware contrastive learning and span filtering that BANER compares against.","marker":"(Li et al., 2023)"},{"why":"It provides ESD, an enhanced span-based decomposition baseline, and the enhanced prototypical-network design.","marker":"(Wang et al., 2022)"},{"why":"It supplies LoRAHub, the dynamic LoRA composition method used to align target-domain information with source-domain knowledge.","marker":"(Huang et al., 2023)"},{"why":"It provides the supervised contrastive learning formulation that the boundary-aware contrastive loss adapts.","marker":"(Khosla et al., 2020)"},{"why":"It supplies the StructShot baseline and the cross-domain evaluation schema where support sets are sampled and the original test set is scored.","marker":"(Yang and Katiyar, 2020)"}],"fun_headline_variants":["Boundary-aware LLM sharpens few-shot NER","BANER: LLM learns entity edges for few-shot NER","LoRAHub + boundary loss boost few-shot NER","Teaching LLMs boundaries lifts few-shot NER"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method's gains rest on the assumption that pulling the entity-type phrase embedding toward the concatenated embeddings of a true span's first and last token, and away from tokens just outside the span, improves how the LLM perceives entity boundaries, and that the hidden layer chosen on the GUM 5-shot set transfers to other domains.","fun_headline_variants_meta":{"raw":{"variants":["Boundary-aware LLM sharpens few-shot NER","BANER: LLM learns entity edges for few-shot NER","LoRAHub + boundary loss boost few-shot NER","Teaching LLMs boundaries lifts few-shot NER"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000201,"raw_usage":{"total_tokens":1381,"prompt_tokens":953,"completion_tokens":428,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":569,"completion_tokens_details":{"reasoning_tokens":360}},"tokens_in":569,"tokens_out":428,"duration_ms":4973,"temperature":1.0,"reasoning_tokens":360,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T23:41:35.581821+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the span-detection stage with and without the boundary-aware contrastive loss under identical LoRA fine-tuning and evaluate on a held-out dataset such as WNUT 5-shot; if removing the loss does not reduce over- and under-detected spans or lower F1, the loss is not doing the claimed work. A second check is to repeat the hidden-layer comparison from Section 4.5 on a different dataset; if the 25th layer is not the best there, the layer choice was overfit to GUM.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"It provides the supervised contrastive learning formulation that the boundary-aware contrastive loss adapts."}],"review_version":1}