{"id":"02e196e2-0505-427e-b4fb-d1a08bfbe585","arxiv_id":"2412.20043","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"STAYKATE blends static representativeness sampling with kNN retrieval to pick in-context examples for LLM scientific NER, reporting modest F1 improvements over KATE and random baselines.","lead":"This paper proposes STAYKATE, a method that selects in-context examples for LLM-based scientific named entity recognition by combining a fixed set of representative samples with per-test retrieval of similar samples. On three science-domain datasets, STAYKATE reports small F1 gains over existing selection methods, but the differences are not backed by statistical tests.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The static component's contribution is supported only by the STAYKATE vs Random+KATE contrast; the static-only ablation shows no benefit and no error bars are reported, so the headline gain may be split noise.","rationale":"The reader's weakest assumption correctly identifies the representativeness heuristic as unvalidated. I agree, but the more precise and load-bearing issue is that the paper itself shows static-only selection does not beat random selection, so the heuristic is validated only indirectly by the hybrid contrast. That contrast lacks error bars and significance testing, and the effect sizes are small relative to the variability induced by three data-pool draws and low-support entity types. The hard-entity improvements, which are the paper's most striking claim, are especially fragile because they depend on very few positive instances. The reader's CONDITIONAL verdict is appropriate: the method is plausible and the results are consistently positive, but the central attribution of the gain to representativeness sampling needs statistical grounding and a clearer ablation before the outperformance claim can be accepted as stated. No change to the reader's verdict is needed.","tokens_in":12162,"tokens_out":4968,"duration_ms":53497,"concrete_test":"Rerun the Table 2 comparison with at least 10 different random data-pool splits (and, if feasible, 3 GPT-3.5 samples per split), computing paired STAYKATE minus Random+KATE micro-F1 differences and a paired bootstrap 95% confidence interval. Also run Representative-only against Random-only on the same splits. If the positive difference is not consistent across splits (e.g., the CI includes zero or fewer than 8 of 10 splits are positive), the static component's contribution is unsupported. As an analytical check, bootstrap the F1 delta for Device (support 45) and Property (support 105) by resampling the roughly 200 test sentences; if the reported delta of about 0.04 is within one standard error, the 'particularly pronounced' claim fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central novelty of STAYKATE is the static representativeness selection from Section 3.2.2, but the paper's own ablation undermines it: in Table 1, Representative (static only) does not outperform Random Sampling on any dataset (MSPT 0.57 vs 0.57, WLP 0.66 vs 0.65, BC5CDR 0.69 vs 0.69; with lambda=1 it is worse on MSPT at 0.54). Thus the only direct evidence that RScore selects better static demonstrations is the hybrid comparison STAYKATE vs Random+KATE in Table 2. That comparison is reported as a mean over three data pools with no variance, confidence intervals, or significance test, and the aggregate differences are small (micro F1 0.61 vs 0.60 on MSPT, 0.69 vs 0.68 on WLP, 0.72 vs 0.70 on BC5CDR). The 'particularly pronounced' hard-entity gains rest on very small positive-instance counts (Property support 105, Device 45, Seal 20); a handful of additional correct predictions changes those F1 scores by several points. Given the stochasticity of GPT-3.5 API calls and the use of only three data-pool draws, the 0.02-0.04 F1 deltas attributed to representativeness could easily arise from split noise. The Section 5.1 statement that 'this highlights the essential role of representativeness sampling' is therefore not yet established. The underlying heuristic, that sentences with near-mean BERT predictive entropy are better demonstrations, receives no independent validation in the paper.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes STAYKATE, a hybrid in-context example selection method for scientific named entity recognition with GPT-3.5. The static component selects a small set of representative examples from an unlabeled pool using a representativeness score based on predictive entropy of a low-resource fine-tuned BERT model (RScore, Eq. 3). The dynamic component retrieves semantically similar examples from a labeled pool using KATE. Experiments on three scientific NER datasets (MSPT, WLP, BC5CDR) compare STAYKATE against fine-tuned BERT, zero-shot prompting, random sampling, static-only representativeness sampling, and KATE. The paper reports micro-F1 and entity-level F1 scores, and claims that STAYKATE outperforms both traditional supervised methods and existing selection methods, with the largest gains on entity types that are otherwise difficult (e.g., Property, Device, Disease). An error analysis attributes the gains to reduced overprediction and improved handling of ambiguous entities.","tokens_in":12532,"tokens_out":4976,"duration_ms":46124,"significance":"If the result is robust, the paper would provide a practical low-resource recipe: a training-free hybrid selection scheme that improves GPT-3.5-based scientific NER beyond fine-tuned BERT and prior ICL selection methods, while also helping hard entity types. The idea of combining a static representative set with dynamic retrieval is intuitive and easy to reproduce, and the error analysis is a useful qualitative contribution. However, the current evidence is not yet strong enough to support the central claim: the static component's contribution is not validated by the static-only ablation, the hybrid gains over Random+KATE are small and reported without variance or significance tests, and several entity-level comparisons rest on very small support counts. The paper would be a solid empirical contribution after substantial additional analysis and reporting.","major_comments":[{"comment":"The ablation evidence does not support the claim that representativeness sampling is essential to the method. In Table 1, the static-only Representative method (lambda=0) ties Random Sampling on MSPT and BC5CDR (0.57 vs 0.57 and 0.69 vs 0.69) and is only 0.01 higher on WLP (0.66 vs 0.65); with lambda=1, it is worse than Random Sampling on MSPT (0.54 vs 0.57). The only direct evidence for the static component is the STAYKATE vs Random+KATE comparison in Table 2, where the micro-F1 deltas are 0.01 on MSPT (0.61 vs 0.60), 0.01 on WLP (0.69 vs 0.68), and 0.02 on BC5CDR (0.72 vs 0.70). These are means over only three data-pool draws, with no standard deviations, confidence intervals, or significance tests. Given the very small test subsets and the stochasticity of the pipeline, the observed deltas could easily arise from split noise. The statement in Section 5.1 that 'this highlights the essential role of representativeness sampling' is therefore not yet established. Please report per-pool results, variance or confidence intervals, and a significance test (e.g., paired bootstrap or approximate randomization) for the hybrid comparisons. Ideally, also provide an independent validation of the RScore heuristic, such as showing that the selected static examples are better than randomly selected static examples when used alone in a larger-scale study.","section":"Section 5.1 / Table 1"},{"comment":"The representativeness assumption underlying RScore is not validated. The method assumes that selecting sentences with predictive entropy H(x) close to mu_H + lambda*sigma_H captures 'the overall patterns of the corpus,' but no evidence is presented that near-mean entropy sentences are indeed better in-context demonstrations. The static-only results in Table 1 are consistent with the null hypothesis that this choice adds no benefit (and can hurt, as with lambda=1 on MSPT). Moreover, lambda is treated as a free parameter and selected per dataset after seeing the results: Section 5.1 states that 'the choice of lambda value depends on the characteristics of the dataset,' and the reported best values are lambda=0 for MSPT and WLP but lambda=1 for BC5CDR. This post-hoc selection inflates the chance of favorable comparisons. Please provide a principled procedure for setting lambda (for example, on a held-out validation set) and report the sensitivity of the conclusions to lambda across a range of values.","section":"Section 3.2.2, Eq. (3)"},{"comment":"The test subsets are small (157-197 sentences) and several entity types have very few gold mentions (Seal support 20, Device 45, Property 105). An entity-level F1 difference of several points can be produced by one or two additional correct predictions on these types. For instance, the Device F1 difference of 0.29 vs 0.25 in Table 2 corresponds to a handful of instances. The abstract's claim that the enhancement is 'particularly pronounced' for hard entity types therefore rests on very small counts. Please report the exact numbers of true positives, false positives, and false negatives for the key comparisons, or provide confidence intervals for the entity-level F1 scores, so that readers can judge whether the differences are meaningful rather than the result of a few instances.","section":"Section 4.1 / Table 5"},{"comment":"The allocation of the total k examples into static (ks) and dynamic (kd) examples is fixed arbitrarily: for k=2, ks=kd=1; for k=6, ks=2 and kd=4; for k=8, ks=2 and kd=6. Because the central claim is that the static component adds value, the paper should demonstrate that the result is not sensitive to the split ratio. Without a sensitivity analysis (e.g., varying ks from 1 to k-1 while keeping k fixed), it is unclear whether the reported improvement of STAYKATE over KATE comes from the specific hybrid ratio, from having any static examples at all, or from the particular representativeness-selected examples. Please add a small experiment varying the ks/kd split, or otherwise justify the chosen allocation.","section":"Section 4.3"}],"minor_comments":[{"comment":"When annotation manuals are not available, entity definitions are generated by GPT-3.5 itself. This could bias the model's behavior in a favorable direction and should be disclosed per dataset. If some datasets use human-authored definitions and others use model-generated definitions, the cross-dataset comparison is not fully controlled; please state which datasets fall into each category.","section":"Section 3.1"},{"comment":"The 'Random + KATE' column is not described in the experimental setup (Section 4.2 or 4.3). Please specify how the random static examples and KATE-retrieved examples were combined, and how many of each were used.","section":"Table 2"},{"comment":"The validation set used for early stopping of BERT is labeled with GPT-3.5 pseudo-labels. This is a non-standard practice and should be justified, since noisy pseudo-labels could affect the fine-tuned BERT model used both for RScore calculation and as a baseline.","section":"Section 4.3"},{"comment":"The sentence 'These results demonstrates that the extraction performance improved as k increases' has a subject-verb agreement error. Also, consider adding a brief note on whether the trend holds for all entity types, as some types (e.g., Seal) do not improve monotonically.","section":"Section 5.1 / Table 4"},{"comment":"The error analysis is presented only as normalized percentages without underlying counts. Adding the raw counts for each error type would make the reductions across methods more interpretable and would help readers assess the magnitude of the observed differences.","section":"Section 6.1 / Figures 5, 8, 9"}],"recommendation":"major_revision","confidential_remarks":"As an editorially oriented note, the paper's core idea is testable and the required additional analyses (variance, significance tests, lambda sensitivity, split sensitivity, small-support accountability) are feasible within a revision. The current manuscript, however, reports only means over three data pools and contains an internal inconsistency between the static-only ablation (no benefit) and the claim that representativeness sampling is essential. I would not reject the paper, but I would require the authors to address the four major comments before considering it for publication. The limited scope (three datasets, one LLM) is acknowledged in the limitations section and should not be a reason for rejection by itself."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Bottom line: STAYKATE is a reasonable hybrid—static representativeness sampling from active learning plus KATE-style retrieval—and the empirical story is consistent across three scientific NER datasets. But the paper's own ablation undermines the central claim that representativeness is what drives the gain. The static-only representative sampling never beats random sampling in Table 1 (sometimes it's worse), so the only direct evidence for RScore is the hybrid comparison against Random+KATE. That comparison is reported as means over three data pools with no variance, confidence intervals, or significance tests, and the test subsets are around 200 sentences. The rare-entity improvements (Property 0.25 vs 0.22, Device 0.29 vs 0.25, Seal 0.48 vs 0.40) rest on support counts of 20–105 instances, so a handful of token predictions shifts those F1 scores. Lambda is chosen per dataset after seeing results, which adds selection bias.\n\nThat said, the paper does real work. The RScore heuristic is new to ICL selection, the combination is clearly motivated, and the experiments cover three diverse domains, multiple k values, and three random pool draws. The error analysis and the case study (the 'solution' example) give a plausible mechanism for why hybrid examples might help. The limitations section is honest about the narrow scope.\n\nThe soft spots are fixable: report error bars, run significance tests, pre-specify lambda or use a validation split, and directly validate the claim that near-mean-entropy sentences are better demonstrations. If those checks hold, the modest 1–3 F1 point gain becomes believable.\n\nWho is this for? Researchers working on low-resource scientific NER or in-context example selection. They'll find a clearly described recipe and a fair comparison against baselines. I'd send it to peer review—it's a legitimate empirical question with a defined method and addressable flaws—but I'd ask the authors to tighten the statistical reporting and soften the claim that representativeness sampling is 'essential.' I agree with the stress-test: the static component's contribution is not established. The paper deserves a serious referee, not a desk rejection.","headline":"Plausible hybrid ICL selector, but the static component's contribution is not actually supported by the paper's own ablation.","tokens_in":13038,"tokens_out":2470,"would_cite":true,"duration_ms":26117,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"STAYKATE, a static-dynamic hybrid example selection method, outperforms fine-tuned BERT and existing selection methods on scientific NER.","keywords":["in-context learning","named entity recognition","example selection","representativeness sampling","retrieval-based selection","scientific information extraction","low-resource NER","STAYKATE"],"falsifier":"An ablation holding the dynamic examples fixed and replacing the mean-entropy static examples with randomly chosen sentences would settle it: if STAYKATE's F1 edge over Random+KATE vanishes, the representativeness score is doing no work. A complementary check is to select static examples from the extremes of the entropy distribution (very low and very high $H(x)$) and verify that performance drops as the near-mean assumption predicts.","tokens_in":12014,"feed_emoji":"🔬","tokens_out":6236,"duration_ms":53838,"temperature":0.7,"pith_summary":"The paper sets out to establish that in-context example selection for large language models does not have to choose between static and dynamic strategies, and that combining them is better than either alone on scientific named entity recognition under low-resource conditions. STAYKATE fixes a small set of 'representative' sentences chosen so their average predictive difficulty matches the corpus, and pairs them with sentences retrieved as the nearest neighbors of each test input. Across three scientific corpora, the reported micro-F1 beats fine-tuned BERT, random sampling, representativeness sampling alone, and KATE alone, with the clearest gains on entity types other methods struggle with, such as Property, Device, and Disease. A sympathetic reader would care because the recipe needs no model retraining and only a handful of human annotations, which matters when labeled scientific data are expensive to produce.","feed_headline":"Hybrid example selection beats fine-tuned BERT for scientific NER","feed_subtitle":"Fixed representative examples plus retrieved neighbors lift GPT-3.5's F1 most on entity types other methods miss","key_machinery":"The load-bearing object is the representativeness score $$RScore = |H(x) - (\\mu_H + \\$\\lambda$ \\sigma_H)|,$$ where $H(x)$ is the mean token-level predictive entropy of a candidate sentence under a fine-tuned BERT model, and $\\mu_H$ and $\\sigma_H$ are the mean and standard deviation of that quantity across the unlabeled pool. Sentences that minimize this score are fixed as static demonstrations, on the assumption that near-average sentences capture the corpus's typical patterns, with $\\lambda$ shifting the preferred zone when the entropy distribution is skewed. The dynamic half retrieves the $k_d$ labeled sentences closest to each test input by cosine similarity of embedding vectors, following KATE. The two sets of examples are concatenated in a single prompt with a domain-expert system role and JSON output instructions.","core_discovery":"The central claim is that the best demonstrations for in-context scientific NER are a hybrid of two complementary kinds of examples, and that the fixed half should come from representativeness sampling rather than random selection. The paper reports micro-F1 gains over the strongest comparison method on all three datasets, and the biggest improvements concentrate on the hardest entity types: Property rises from 0.22 with KATE to 0.25 with STAYKATE, Device from 0.25 to 0.29, and Disease from 0.67 to 0.70. Error analysis attributes the gains to fewer overprediction errors and better handling of words whose entity type depends on context, such as 'solution' appearing as either a Material or a Property.","pith_inferences":["A natural extension the paper does not run: replace the fine-tuned BERT entropy signal with a cheaper frozen-model signal, and test whether the near-mean rule transfers to other extraction tasks such as relation extraction or event extraction.","The $\\lambda$ dependence tracks the share of non-entity tokens, so a testable refinement would predict $\\lambda$ from that ratio instead of tuning it per dataset.","The WLP finding that identical sentences sometimes carry inconsistent annotations suggests a stress test: datasets with high annotation disagreement should amplify the contrast between KATE and STAYKATE if the disambiguation story is right.","Porting the recipe to open-weight models would show whether the selection method transfers beyond the commercial model used in the paper."],"forward_implications":["Under low-resource scientific conditions, GPT-3.5 with STAYKATE surpasses fine-tuned BERT, making prompt-based extraction a viable alternative to supervised fine-tuning.","The largest F1 gains land on infrequent or context-sensitive entity types such as Property, Device, and Disease, where retrieval alone leaves the model guessing.","The method works with as few as two total demonstrations, so the extra human annotation burden for the static examples stays small.","The drop from Random+KATE to STAYKATE is evidence that the fixed examples must be representative, not merely present, for the hybrid to help."],"supporting_citations":[{"why":"Supplies KATE, the retrieval-based dynamic selection method that STAYKATE extends and compares against.","marker":"Liu et al. (2022)"},{"why":"Provides the diversity-and-uncertainty selection idea behind the representativeness score formula.","marker":"Kumar et al. (2022)"},{"why":"Frames active-learning representativeness sampling for BERT, the motivation for the static selection design.","marker":"Ein-Dor et al. (2020)"},{"why":"Introduces in-context learning and the few-shot GPT paradigm that the whole method operates within.","marker":"Brown et al. (2020)"},{"why":"Provides the MSPT materials science corpus used as one of the three evaluation datasets.","marker":"Mysore et al. (2019)"},{"why":"Provides the WLP wet-lab protocol corpus used as one of the three evaluation datasets.","marker":"Kulkarni et al. (2018)"},{"why":"Provides the BC5CDR biomedical corpus used as one of the three evaluation datasets.","marker":"Li et al. (2016)"},{"why":"Supplies BERT, both the fine-tuned supervised baseline and the model used to compute predictive entropy.","marker":"Devlin et al. (2019)"},{"why":"Supports the domain-expert system role template used in prompt construction.","marker":"Peng et al. (2023)"}],"fun_headline_variants":["Hybrid demo selection boosts GPT-3.5 NER on hard entity types","Hybrid in-context examples lift NER F1 most on tricky science terms","Representative plus retrieved demos win for scientific NER","Hybrid prompt examples best for scientific entity extraction","STAYKATE hybrid demos sharpen scientific NER on hard terms"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The static component rests on the premise that sentences whose BERT predictive entropy is near the corpus mean are representative enough to make good fixed demonstrations, an assumption the paper validates only indirectly through overall results rather than by an independent check.","fun_headline_variants_meta":{"raw":{"variants":["Hybrid demo selection boosts GPT-3.5 NER on hard entity types","Hybrid in-context examples lift NER F1 most on tricky science terms","Representative plus retrieved demos win for scientific NER","Hybrid prompt examples best for scientific entity extraction","STAYKATE hybrid demos sharpen scientific NER on hard terms"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000491,"raw_usage":{"total_tokens":2358,"prompt_tokens":832,"completion_tokens":1526,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":448,"completion_tokens_details":{"reasoning_tokens":1434}},"tokens_in":448,"tokens_out":1526,"duration_ms":10528,"temperature":1.0,"reasoning_tokens":1434,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T23:36:45.135083+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"An ablation holding the dynamic examples fixed and replacing the mean-entropy static examples with randomly chosen sentences would settle it: if STAYKATE's F1 edge over Random+KATE vanishes, the representativeness score is doing no work. A complementary check is to select static examples from the extremes of the entropy distribution (very low and very high $H(x)$) and verify that performance drops as the near-mean assumption predicts.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces in-context learning and the few-shot GPT paradigm that the whole method operates within."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the MSPT materials science corpus used as one of the three evaluation datasets."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the WLP wet-lab protocol corpus used as one of the three evaluation datasets."}],"review_version":1}