Pith. sign in

REVIEW 4 major objections 5 minor 36 references

STAYKATE: Hybrid In-Context Example Selection Combining Representativeness Sampling and Retrieval-based Approach -- A Case Study on Science Domains

T0 review · 4 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read STAYKATE, a static-dynamic hybrid example selection method, outperforms fine-tuned BERT and existing selection methods on scientific NER.

desk verdict Plausible hybrid ICL selector, but the static component's contribution is not actually supported by the paper's own ablation. read the letter →

arxiv 2412.20043 v1 pith:QEN7FRZH submitted 2024-12-28 cs.CL

Chencheng Zhu , Kazutaka Shimada , Tomoki Taniguchi , Tomoko Ohkuma This is my paper · ORCID
classification cs.CL
keywords in-contextlearningnamedentityrecognitionexampleselectionrepresentativenesssamplingretrieval-basedscientificinformationextractionlow-resourceNERSTAYKATE
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to establish that in-context example selection for large language models does not have to choose between static and dynamic strategies, and that combining them is better than either alone on scientific named entity recognition under low-resource conditions. STAYKATE fixes a small set of 'representative' sentences chosen so their average predictive difficulty matches the corpus, and pairs them with sentences retrieved as the nearest neighbors of each test input. Across three scientific corpora, the reported micro-F1 beats fine-tuned BERT, random sampling, representativeness sampling alone, and KATE alone, with the clearest gains on entity types other methods struggle with, such as Property, Device, and Disease. A sympathetic reader would care because the recipe needs no model retraining and only a handful of human annotations, which matters when labeled scientific data are expensive to produce.

What carries the argument

The load-bearing object is the representativeness score $$RScore = |H(x) - (\mu_H + \$\lambda$ \sigma_H)|,$$ where $H(x)$ is the mean token-level predictive entropy of a candidate sentence under a fine-tuned BERT model, and $\mu_H$ and $\sigma_H$ are the mean and standard deviation of that quantity across the unlabeled pool. Sentences that minimize this score are fixed as static demonstrations, on the assumption that near-average sentences capture the corpus's typical patterns, with $\lambda$ shifting the preferred zone when the entropy distribution is skewed. The dynamic half retrieves the $k_d$ labeled sentences closest to each test input by cosine similarity of embedding vectors, following KATE. The two sets of examples are concatenated in a single prompt with a domain-expert system role and JSON output instructions.

What would settle it

An ablation holding the dynamic examples fixed and replacing the mean-entropy static examples with randomly chosen sentences would settle it: if STAYKATE's F1 edge over Random+KATE vanishes, the representativeness score is doing no work. A complementary check is to select static examples from the extremes of the entropy distribution (very low and very high $H(x)$) and verify that performance drops as the near-mean assumption predicts.

Watch

Extended reading notes

Core claim

The central claim is that the best demonstrations for in-context scientific NER are a hybrid of two complementary kinds of examples, and that the fixed half should come from representativeness sampling rather than random selection. The paper reports micro-F1 gains over the strongest comparison method on all three datasets, and the biggest improvements concentrate on the hardest entity types: Property rises from 0.22 with KATE to 0.25 with STAYKATE, Device from 0.25 to 0.29, and Disease from 0.67 to 0.70. Error analysis attributes the gains to fewer overprediction errors and better handling of words whose entity type depends on context, such as 'solution' appearing as either a Material or a Property.

Load-bearing premise

The static component rests on the premise that sentences whose BERT predictive entropy is near the corpus mean are representative enough to make good fixed demonstrations, an assumption the paper validates only indirectly through overall results rather than by an independent check.

Editorial extensions

If this is right

  • Under low-resource scientific conditions, GPT-3.5 with STAYKATE surpasses fine-tuned BERT, making prompt-based extraction a viable alternative to supervised fine-tuning.
  • The largest F1 gains land on infrequent or context-sensitive entity types such as Property, Device, and Disease, where retrieval alone leaves the model guessing.
  • The method works with as few as two total demonstrations, so the extra human annotation burden for the static examples stays small.
  • The drop from Random+KATE to STAYKATE is evidence that the fixed examples must be representative, not merely present, for the hybrid to help.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural extension the paper does not run: replace the fine-tuned BERT entropy signal with a cheaper frozen-model signal, and test whether the near-mean rule transfers to other extraction tasks such as relation extraction or event extraction.
  • The $\lambda$ dependence tracks the share of non-entity tokens, so a testable refinement would predict $\lambda$ from that ratio instead of tuning it per dataset.
  • The WLP finding that identical sentences sometimes carry inconsistent annotations suggests a stress test: datasets with high annotation disagreement should amplify the contrast between KATE and STAYKATE if the disambiguation story is right.
  • Porting the recipe to open-weight models would show whether the selection method transfers beyond the commercial model used in the paper.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes STAYKATE, a hybrid in-context example selection method for scientific named entity recognition with GPT-3.5. The static component selects a small set of representative examples from an unlabeled pool using a representativeness score based on predictive entropy of a low-resource fine-tuned BERT model (RScore, Eq. 3). The dynamic component retrieves semantically similar examples from a labeled pool using KATE. Experiments on three scientific NER datasets (MSPT, WLP, BC5CDR) compare STAYKATE against fine-tuned BERT, zero-shot prompting, random sampling, static-only representativeness sampling, and KATE. The paper reports micro-F1 and entity-level F1 scores, and claims that STAYKATE outperforms both traditional supervised methods and existing selection methods, with the largest gains on entity types that are otherwise difficult (e.g., Property, Device, Disease). An error analysis attributes the gains to reduced overprediction and improved handling of ambiguous entities.

Significance. If the result is robust, the paper would provide a practical low-resource recipe: a training-free hybrid selection scheme that improves GPT-3.5-based scientific NER beyond fine-tuned BERT and prior ICL selection methods, while also helping hard entity types. The idea of combining a static representative set with dynamic retrieval is intuitive and easy to reproduce, and the error analysis is a useful qualitative contribution. However, the current evidence is not yet strong enough to support the central claim: the static component's contribution is not validated by the static-only ablation, the hybrid gains over Random+KATE are small and reported without variance or significance tests, and several entity-level comparisons rest on very small support counts. The paper would be a solid empirical contribution after substantial additional analysis and reporting.

major comments (4)
  1. [Section 5.1 / Table 1] The ablation evidence does not support the claim that representativeness sampling is essential to the method. In Table 1, the static-only Representative method (lambda=0) ties Random Sampling on MSPT and BC5CDR (0.57 vs 0.57 and 0.69 vs 0.69) and is only 0.01 higher on WLP (0.66 vs 0.65); with lambda=1, it is worse than Random Sampling on MSPT (0.54 vs 0.57). The only direct evidence for the static component is the STAYKATE vs Random+KATE comparison in Table 2, where the micro-F1 deltas are 0.01 on MSPT (0.61 vs 0.60), 0.01 on WLP (0.69 vs 0.68), and 0.02 on BC5CDR (0.72 vs 0.70). These are means over only three data-pool draws, with no standard deviations, confidence intervals, or significance tests. Given the very small test subsets and the stochasticity of the pipeline, the observed deltas could easily arise from split noise. The statement in Section 5.1 that 'this highlights the essential role of representativeness sampling' is therefore not yet established. Please report per-pool results, variance or confidence intervals, and a significance test (e.g., paired bootstrap or approximate randomization) for the hybrid comparisons. Ideally, also provide an independent validation of the RScore heuristic, such as showing that the selected static examples are better than randomly selected static examples when used alone in a larger-scale study.
  2. [Section 3.2.2, Eq. (3)] The representativeness assumption underlying RScore is not validated. The method assumes that selecting sentences with predictive entropy H(x) close to mu_H + lambda*sigma_H captures 'the overall patterns of the corpus,' but no evidence is presented that near-mean entropy sentences are indeed better in-context demonstrations. The static-only results in Table 1 are consistent with the null hypothesis that this choice adds no benefit (and can hurt, as with lambda=1 on MSPT). Moreover, lambda is treated as a free parameter and selected per dataset after seeing the results: Section 5.1 states that 'the choice of lambda value depends on the characteristics of the dataset,' and the reported best values are lambda=0 for MSPT and WLP but lambda=1 for BC5CDR. This post-hoc selection inflates the chance of favorable comparisons. Please provide a principled procedure for setting lambda (for example, on a held-out validation set) and report the sensitivity of the conclusions to lambda across a range of values.
  3. [Section 4.1 / Table 5] The test subsets are small (157-197 sentences) and several entity types have very few gold mentions (Seal support 20, Device 45, Property 105). An entity-level F1 difference of several points can be produced by one or two additional correct predictions on these types. For instance, the Device F1 difference of 0.29 vs 0.25 in Table 2 corresponds to a handful of instances. The abstract's claim that the enhancement is 'particularly pronounced' for hard entity types therefore rests on very small counts. Please report the exact numbers of true positives, false positives, and false negatives for the key comparisons, or provide confidence intervals for the entity-level F1 scores, so that readers can judge whether the differences are meaningful rather than the result of a few instances.
  4. [Section 4.3] The allocation of the total k examples into static (ks) and dynamic (kd) examples is fixed arbitrarily: for k=2, ks=kd=1; for k=6, ks=2 and kd=4; for k=8, ks=2 and kd=6. Because the central claim is that the static component adds value, the paper should demonstrate that the result is not sensitive to the split ratio. Without a sensitivity analysis (e.g., varying ks from 1 to k-1 while keeping k fixed), it is unclear whether the reported improvement of STAYKATE over KATE comes from the specific hybrid ratio, from having any static examples at all, or from the particular representativeness-selected examples. Please add a small experiment varying the ks/kd split, or otherwise justify the chosen allocation.
minor comments (5)
  1. [Section 3.1] When annotation manuals are not available, entity definitions are generated by GPT-3.5 itself. This could bias the model's behavior in a favorable direction and should be disclosed per dataset. If some datasets use human-authored definitions and others use model-generated definitions, the cross-dataset comparison is not fully controlled; please state which datasets fall into each category.
  2. [Table 2] The 'Random + KATE' column is not described in the experimental setup (Section 4.2 or 4.3). Please specify how the random static examples and KATE-retrieved examples were combined, and how many of each were used.
  3. [Section 4.3] The validation set used for early stopping of BERT is labeled with GPT-3.5 pseudo-labels. This is a non-standard practice and should be justified, since noisy pseudo-labels could affect the fine-tuned BERT model used both for RScore calculation and as a baseline.
  4. [Section 5.1 / Table 4] The sentence 'These results demonstrates that the extraction performance improved as k increases' has a subject-verb agreement error. Also, consider adding a brief note on whether the trend holds for all entity types, as some types (e.g., Seal) do not improve monotonically.
  5. [Section 6.1 / Figures 5, 8, 9] The error analysis is presented only as normalized percentages without underlying counts. Adding the raw counts for each error type would make the reductions across methods more interpretable and would help readers assess the magnitude of the observed differences.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: STAYKATE's selection rules are defined independently of the gold labels and the claimed gains are evaluated externally; no equation reduces to its inputs.

full rationale

The paper's central comparison (STAYKATE vs KATE vs random vs BERT) is an empirical evaluation against held-out gold annotations. The static selection score RScore = |H(x) - (mu_H + lambda*sigma_H)| is computed from a BERT model's predictive entropy on unlabeled sentences; the entropy is not derived from the F1 values it is later used to predict. The dynamic KATE retrieval uses cosine similarity on OpenAI embeddings, again independent of the target metric. There is no fitted parameter that is renamed as a prediction, and no load-bearing self-citation: the paper's references to prior work (e.g., Kumar et al. 2022 for RScore, Liu et al. 2022 for KATE) are external sources, not the authors' own unverified results. The use of GPT-3.5 to generate entity definitions when no annotation manual is available is a potential methodological weakness, but it applies equally to all compared methods and does not make the STAYKATE-vs-baseline comparison circular. Likewise, the choice of lambda based on dataset characteristics and the small support counts for rare entities are robustness concerns, not cases where an output is equivalent to an input by construction. Therefore no circular step can be exhibited; score 0.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The central claim rests on two domain heuristics (entropy representativeness and embedding retrieval) and two hand-set hyperparameters (lambda and the ks/kd split), but no new theoretical entities are introduced.

free parameters (2)
  • lambda (RScore zone parameter) = 0 or 1 chosen per dataset
    Controls the distance from the mean predictive entropy in Eq. 3. The paper reports results for lambda=0 and lambda=1 and highlights the better per dataset (Table 1) without a pre-specified selection rule.
  • ks/kd split = e.g., ks=2, kd=6 for k=8
    Set by hand for each total k; no search or justification is given beyond allocating static and dynamic examples.
assumptions (3)
  • domain assumption Near-mean predictive entropy marks representative demonstrations for ICL
    Section 3.2.2: "It is assumed that the selection around mu_H can represent the overall patterns of the corpus."
  • domain assumption Embedding-space cosine similarity retrieves useful demonstrations
    Section 3.3 inherits KATE's assumption that nearest neighbors in OpenAI embedding space are helpful ICL examples.
  • domain assumption GPT-3.5-generated entity definitions are adequate when the annotation manual is missing
    Section 3.1: "otherwise, we ask GPT-3.5 to generate definitions based on its own understanding."

how reviews work

0 comments
Cite this review

Pith. "Pith review of STAYKATE: Hybrid In-Context Example Selection Combining Representativeness Sampling and Retrieval-based Approach -- A Case Study on Science Domains." pith.science (2026). https://pith.science/paper/QEN7FRZH

@misc{pith2026241220043,
  author       = {Pith},
  title        = {Pith review of: STAYKATE: Hybrid In-Context Example Selection Combining Representativeness Sampling and Retrieval-based Approach -- A Case Study on Science Domains},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QEN7FRZH}},
  note         = {Machine review of arXiv:2412.20043}
}
read the original abstract

Large language models (LLMs) demonstrate the ability to learn in-context, offering a potential solution for scientific information extraction, which often contends with challenges such as insufficient training data and the high cost of annotation processes. Given that the selection of in-context examples can significantly impact performance, it is crucial to design a proper method to sample the efficient ones. In this paper, we propose STAYKATE, a static-dynamic hybrid selection method that combines the principles of representativeness sampling from active learning with the prevalent retrieval-based approach. The results across three domain-specific datasets indicate that STAYKATE outperforms both the traditional supervised methods and existing selection methods. The enhancement in performance is particularly pronounced for entity types that other methods pose challenges.

Figures

Figures reproduced from arXiv: 2412.20043 by the authors.

Figure 1
Figure 1. The overall process of STAYKATE. The right side of the figure shows the entire prompt structure. [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Our experimental setting about data pool. [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. The distribution of predictive entropy for each dataset. [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Statistics on the percentage of various error [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: Statistics of errors across different selection [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: A case study comparing KATE with STAYKATE. [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 7
Figure 7. Figure 7: Sentence pair with inconsistent annotation in WLP. [PITH_FULL_IMAGE:figures/full_fig_p011_7.png]
Figure 8
Figure 8. Figure 8: Statistics of errors across different selection [PITH_FULL_IMAGE:figures/full_fig_p011_8.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

36 extracted references · 8 canonical work pages

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Emily Alsentzer, John Murphy, William Boag, Wei-Hung Weng, Di Jindi, Tristan Naumann, and Matthew McDermott. 2019. https://doi.org/10.18653/v1/W19-1909 Publicly available clinical BERT embeddings . In Proceedings of the 2nd Clinical Natural Language Processing Workshop, pages 72--78

    doi:10.18653/v1/w19-1909 2019

  4. [4]

    Iz Beltagy, Kyle Lo, and Arman Cohan. 2019. https://doi.org/10.18653/v1/D19-1371 S ci BERT : A pretrained language model for scientific text . In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3615--3620

    doi:10.18653/v1/d19-1371 2019

  5. [5]

    Terra Blevins, Hila Gonen, and Luke Zettlemoyer. 2022. https://arxiv.org/abs/2211.07830 Prompting language models for linguistic structure . Preprint, arXiv:2211.07830

    arXiv 2022

  6. [6]

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, and Amanda et al Askell. 2020. https://proceedings.neurips.cc/paper_files/paper/2020/file/1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf Language models are few-shot learners . In Advances in Neural Information Processing ...

    work page 2020

  7. [7]

    Jiawei Chen, Yaojie Lu, Hongyu Lin, Jie Lou, Wei Jia, Dai Dai, Hua Wu, Boxi Cao, Xianpei Han, and Le Sun. 2023. https://doi.org/10.18653/v1/2023.acl-long.764 Learning in-context learning for named entity recognition . In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 13661--13675

    doi:10.18653/v1/2023.acl-long.764 2023

  8. [8]

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. https://doi.org/10.18653/v1/N19-1423 BERT : Pre-training of deep bidirectional transformers for language understanding . In Proceedings of the 2019 Conference of the North A merican Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long a...

    doi:10.18653/v1/n19-1423 2019

Show all 36 references
  1. [9]

    Bosheng Ding, Chengwei Qin, Linlin Liu, Yew Ken Chia, Shafiq Joty, Boyang Li, and Lidong Bing. 2023. Is GPT -3 a good data annotator? In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 11173--11195

    2023

  2. [10]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, and Angela Fan et al. 2024. https://arxiv.org/abs/2407.21783 The llama 3 herd of models . Preprint, arXiv:2407.21783

    2024 arXiv

  3. [11]

    Liat Ein-Dor, Alon Halfon, Ariel Gera, Eyal Shnarch, Lena Dankin, Leshem Choshen, Marina Danilevsky, Ranit Aharonov, Yoav Katz, and Noam Slonim. 2020. https://doi.org/10.18653/v1/2020.emnlp-main.638 A ctive L earning for BERT : A n E mpirical S tudy . In Proceedings of the 202...

    2020 doi

  4. [12]

    Fabrizio Gilardi, Meysam Alizadeh, and Maël Kubli. 2023. https://doi.org/10.1073/pnas.2305016120 Chatgpt outperforms crowd workers for text-annotation tasks . Proceedings of the National Academy of Sciences, 120(30):e2305016120

    2023 doi

  5. [13]

    Zhi Hong, Logan Ward, Kyle Chard, Ben Blaiszik, and Ian Foster. 2021. https://doi.org/10.1007/s11837-021-04902-9 Challenges and advances in information extraction from scientific literature: a review . JOM, 73(11):3383--3400

    2021 doi

  6. [14]

    Bernal Jimenez Gutierrez, Nikolas McNeal, Clayton Washington, You Chen, Lang Li, Huan Sun, and Yu Su. 2022. https://doi.org/10.18653/v1/2022.findings-emnlp.329 Thinking about GPT -3 in-context learning for biomedical IE ? think again . In Findings of the Association for Comput...

    2022 doi

  7. [15]

    Chaitanya Kulkarni, Wei Xu, Alan Ritter, and Raghu Machiraju. 2018. https://doi.org/10.18653/v1/N18-2016 An annotated corpus for machine reading of instructions in wet lab protocols . In Proceedings of the 2018 Conference of the North A merican Chapter of the Association for C...

    2018 doi

  8. [16]

    Shanu Kumar, Sandipan Dandapat, and Monojit Choudhury. 2022. https://doi.org/10.18653/v1/2022.findings-naacl.78 '' diversity and uncertainty in moderation '' are the key to data selection for multilingual few-shot transfer . In Findings of the Association for Computational Lin...

    2022 doi

  9. [17]

    Md Tahmid Rahman Laskar, M Saiful Bari, Mizanur Rahman, Md Amran Hossen Bhuiyan, Shafiq Joty, and Jimmy Huang. 2023. https://doi.org/10.18653/v1/2023.findings-acl.29 A systematic study and comprehensive evaluation of C hat GPT on benchmark datasets . In Findings of the Associa...

    2023 doi

  10. [18]

    Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang. 2019. https://doi.org/10.1093/bioinformatics/btz682 BioBERT: a pre-trained biomedical language representation model for biomedical text mining . Bioinformatics, 36(4):1234--1240

    2019 doi

  11. [19]

    Jiao Li, Yueping Sun, Robin J Johnson, Daniela Sciaky, Chih-Hsuan Wei, Robert Leaman, Allan Peter Davis, Carolyn J Mattingly, Thomas C Wiegers, and Zhiyong Lu. 2016. BioCreative V CDR task corpus: a resource for chemical disease relation extraction. Database (Oxford), 2016:baw068

    2016

  12. [20]

    Hongfu Liu and Ye Wang. 2023. https://doi.org/10.18653/v1/2023.findings-emnlp.1060 Towards informative few-shot prompt with maximum information gain for in-context learning . In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 15825--15838, Singapore

    2023 doi

  13. [21]

    Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, and Weizhu Chen. 2022. https://doi.org/10.18653/v1/2022.deelio-1.10 What makes good in-context examples for GPT -3? In Proceedings of Deep Learning Inside Out (DeeLIO 2022): The 3rd Workshop on Knowledge Extr...

    2022 doi

  14. [22]

    Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. 2022. https://doi.org/10.18653/v1/2022.acl-long.556 Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity . In Proceedings of the 60th Annual Meeting of th...

    2022 doi

  15. [23]

    Masoud Monajatipoor, Jiaxin Yang, Joel Stremmel, Melika Emami, Fazlolah Mohaghegh, Mozhdeh Rouhsedaghat, and Kai-Wei Chang. 2024. https://arxiv.org/abs/2404.07376 Llms in biomedicine: A study on clinical named entity recognition . Preprint, arXiv:2404.07376

    2024 arXiv

  16. [24]

    Sheshera Mysore, Zachary Jensen, Edward Kim, Kevin Huang, Haw-Shiuan Chang, Emma Strubell, Jeffrey Flanigan, Andrew McCallum, and Elsa Olivetti. 2019. https://doi.org/10.18653/v1/W19-4007 The materials science procedural text corpus: Annotating materials synthesis procedures w...

    2019 doi

  17. [25]

    Harsha Nori, Yin Tat Lee, Sheng Zhang, Dean Carignan, Richard Edgar, Nicolo Fusi, Nicholas King, Jonathan Larson, Yuanzhi Li, and Weishung Liu et al. 2023. https://arxiv.org/abs/2311.16452 Can generalist foundation models outcompete special-purpose tuning? case study in medici...

    2023 arXiv

  18. [26]

    Keqin Peng, Liang Ding, Qihuang Zhong, Li Shen, Xuebo Liu, Min Zhang, Yuanxin Ouyang, and Dacheng Tao. 2023. https://doi.org/10.18653/v1/2023.findings-emnlp.373 Towards making the most of C hat GPT for machine translation . In Findings of the Association for Computational Ling...

    2023 doi

  19. [27]

    Ohad Rubin, Jonathan Herzig, and Jonathan Berant. 2022. https://doi.org/10.18653/v1/2022.naacl-main.191 Learning to retrieve prompts for in-context learning . In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: ...

    2022 doi

  20. [28]

    Sara Mahdavi, Jason Wei, Hyung Won Chung, Nathan Scales, Ajay Tanwani, Heather Cole-Lewis, and Stephen Pfohl et al

    Karan Singhal, Shekoofeh Azizi, Tao Tu, S. Sara Mahdavi, Jason Wei, Hyung Won Chung, Nathan Scales, Ajay Tanwani, Heather Cole-Lewis, and Stephen Pfohl et al. 2022. https://arxiv.org/abs/2212.13138 Large language models encode clinical knowledge . Preprint, arXiv:2212.13138

    2022 arXiv

  21. [29]

    Yu Song, Santiago Miret, and Bang Liu. 2023. https://doi.org/10.18653/v1/2023.acl-long.201 M at S ci- NLP : Evaluating scientific language models on materials science language tasks using text-to-schema modeling . In Proceedings of the 61st Annual Meeting of the Association fo...

    2023 doi

  22. [30]

    Gupta Tanishq, Zaki Mohd, and N. M. Krishnan. 2022. https://doi.org/10.1038/s41524-022-00784-w Matscibert: A materials domain language model for text mining and information extraction . Npj Computational Materials, 8(1):1--11

    2022 doi

  23. [31]

    Persson, Gerbrand Ceder, and Anubhav Jain

    Amalie Trewartha, Nicholas Walker, Haoyan Huo, Sanghoon Lee, Kevin Cruse, John Dagdelen, Alexander Dunn, Kristin A. Persson, Gerbrand Ceder, and Anubhav Jain. 2022. https://doi.org/10.1016/j.patter.2022.100488 Quantifying the advantage of domain-specific pre-training on named ...

    2022

  24. [32]

    Xingchen Wan, Ruoxi Sun, Hanjun Dai, Sercan Arik, and Tomas Pfister. 2023 a . https://doi.org/10.18653/v1/2023.findings-acl.216 Better zero-shot reasoning with self-adaptive prompting . In Findings of the Association for Computational Linguistics: ACL 2023, pages 3493--3514

    2023 doi

  25. [33]

    Zhen Wan, Fei Cheng, Zhuoyuan Mao, Qianying Liu, Haiyue Song, Jiwei Li, and Sadao Kurohashi. 2023 b . https://doi.org/10.18653/v1/2023.emnlp-main.214 GPT - RE : In-context learning for relation extraction using large language models . In Proceedings of the 2023 Conference on E...

    2023 doi

  26. [34]

    Taiki Watanabe, Akihiro Tamura, Takashi Ninomiya, Takuya Makino, and Tomoya Iwakura. 2019. https://doi.org/10.18653/v1/D19-1648 Multi-task learning for chemical named entity recognition with chemical compound paraphrasing . In Proceedings of the 2019 Conference on Empirical Me...

    2019 doi

  27. [35]

    Persson, Gerbrand Ceder, and Anubhav Jain

    Leigh Weston, Vahe Tshitoyan, John Dagdelen, Olga Vitalievna Kononova, Amalie Trewartha, Kristin A. Persson, Gerbrand Ceder, and Anubhav Jain. 2019. https://doi.org/10.1021/acs.jcim.9b00470 Named entity recognition and normalization applied to large-scale information extractio...

    2019 doi

  28. [36]

    Yiming Zhang, Shi Feng, and Chenhao Tan. 2022. https://doi.org/10.18653/v1/2022.emnlp-main.622 Active example selection for in-context learning . In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 9134--9148, Abu Dhabi, United Arab...

    2022 doi

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.