Pith. sign in

REVIEW 3 major objections 4 minor 42 references

LLMs as Data Annotators: How Close Are We to Human Performance

T0 review · 3 major / 4 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Retrieval-augmented prompting consistently improves LLM data annotation for named entity recognition, closing most of the gap to human-level F1 on structured datasets.

desk verdict A useful but overclaimed benchmark of LLM-as-annotator; the undocumented Human baseline and inconsistent RAG-vs-ICL results need fixing before the title claim holds. read the letter →

arxiv 2504.15022 v1 pith:ZP2YH5SC submitted 2025-04-21 cs.CL

Muhammad Uzair Ul Haq , Davide Rigoni , Alessandro Sperduti This is my paper · ORCID
classification cs.CL
keywords LLMdataannotationNamedEntityRecognitionRetrieval-AugmentedGenerationIn-contextlearningtokenclassificationhuman-levelembeddingmodelsNERevaluation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper asks how close large language models are to human performance when they annotate named entities, and it argues that the gap is much smaller when the context examples are retrieved automatically rather than sampled at random. Across four NER datasets of increasing difficulty, retrieval-augmented generation (RAG) consistently beats both zero-shot and in-context learning, bringing the best models to within about 1 to 3 F1 points of human-level annotation on CoNLL-2003, WNUT-17, and GUM. The exception is SKILLSPAN, a soft-skill extraction dataset where the best LLM still trails human F1 by roughly 20 points. If this holds, retrieval-based context selection is the key lever for making LLM annotation cheap and useful, and the main research frontier moves to hard, ambiguous entity types.

What carries the argument

The central object is retrieval-augmented generation (RAG) applied to prompt construction: each sentence to be annotated is embedded, and the $M$ most similar human-annotated examples are pulled from a vector store to form the in-context context. This replaces the random sample of context examples used by ICL, giving the LLM examples that are semantically close to the input. The other load-bearing piece is structured output formatting, which the paper uses to keep token-label correspondence intact for gpt-4o-mini; it notes that open-weight models still show output inconsistencies.

What would settle it

Recompute the human baselines under the paper's own protocol: fine-tune RoBERTa on a 30% human-annotated subset with the same five seeds and report F1. If those numbers are lower than 92.12, 54.93, 58.26, and 54.79, the claimed gaps (2.7%, 1.21%, 3.15%) are underestimates of the true gap to human annotation.

Watch

Extended reading notes

Core claim

The paper's central claim is that for token-level NER annotation, a RAG-based method—embedding each sentence, retrieving the most similar human-annotated examples from a vector store, and placing them in the prompt—consistently outperforms randomly sampled ICL and zero-shot prompting. On CoNLL-2003, gpt-4o-mini with OpenAI embeddings reaches 89.72 F1 with 75 retrieved examples, within 2.7 points of the 92.12 human score; on WNUT-17, Qwen2.5-72B reaches 53.72 versus 54.93 human; on GUM, Qwen2.5-72B reaches 55.11 versus 58.26 human. The paper also reports that with the right retriever, 7B-parameter models can be statistically indistinguishable from 70B models, so model size is not the only driver. On SKILLSPAN, however, the best F1 is 34.06 versus 54.79 human, showing the method does not solve semantically ambiguous entities.

Load-bearing premise

The entire 'gap to human' comparison rests on an unstated human baseline: the paper does not say how the Human rows in Tables 2-4 were computed, and if they were not produced by the same RoBERTa-on-30%-subset protocol, the stated gaps are not valid comparisons.

Editorial extensions

If this is right

  • LLM annotation with retrieval is production-usable for well-structured entity types: CoNLL-2003, WNUT-17, and GUM all land within 1–3 F1 points of the human baseline.
  • Context selection matters as much as model scale: 7B models paired with strong embeddings can match or approach 70B models, so cost-conscious annotation can use smaller models.
  • Randomly sampled ICL is a weak default; a vector-store retriever is a cheap, consistent upgrade that needs no additional human effort.
  • The method does not transfer to soft-skill extraction: SKILLSPAN remains roughly 20 F1 points below human, so progress on ambiguous, multi-token entity types needs another idea.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A reader should treat the precise gap percentages as provisional: the paper never states how the 'Human' rows in Tables 2-4 were produced, so if those numbers come from full-training-set models, the true gap to a same-protocol human baseline would be larger.
  • If retrieval quality is the main lever, then a testable extension is to swap the dense retriever for hybrid or re-ranked retrieval and measure annotation F1; the paper uses a 'naive' RAG by its own admission.
  • The same retrieval principle likely transfers to other token-level labeling tasks such as aspect or event extraction, where similar-example context could reduce annotation cost; this is a conjecture, not a paper claim.
  • On SKILLSPAN, the failure mode may be the embedding space itself: dense embeddings built for sentence similarity do not represent abstract soft-skill phrases, so a dedicated skill-aware retriever is a natural next test.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper evaluates the use of LLMs as automatic annotators for named entity recognition (NER), comparing a zero-shot baseline, in-context learning (ICL) with randomly sampled context examples, and a retrieval-augmented generation (RAG) approach in which context examples are retrieved by cosine similarity from a vector store using one of two embedding models. Five LLMs (gpt-4o-mini, Qwen2.5-72B, Llama3.5-70B, Qwen2.5-7B, and Llama3.1-8B) are assessed on CoNLL-2003, WNUT-17, GUM, and SKILLSPAN. Annotation quality is measured by fine-tuning RoBERTa on the LLM-annotated portion of the training set and reporting F1 on the test set, with results averaged over five seeds. The paper reports that RAG-based methods outperform ICL and the baseline, that gpt-4o-mini and Qwen2.5-72B come within a few F1 points of human-level annotation on CoNLL-2003, and that SKILLSPAN remains far below human performance.

Significance. If the results hold, the paper would provide useful practical guidance on when LLM-based NER annotation can substitute for human annotation and whether retrieval-based context selection is the main lever. The study has notable strengths: five LLMs at two parameter scales, three context sizes, two embedding models, four datasets of differing complexity, seed-averaged results with standard deviations, a non-parametric Friedman/Conover analysis, and qualitative examples of model outputs. The evaluation is a genuine measurement on held-out test sets, so circularity is not an issue. However, the headline 'gap to human' claims rest on an undocumented human baseline, the 30% training-sample protocol is underspecified, and the universal claim that RAG consistently outperforms ICL is contradicted by several cells in the paper's own tables. These are load-bearing issues for the central conclusions, though they are potentially fixable in revision.

major comments (3)
  1. [§4.2, §4.3, Tables 2–4] The Human rows in Tables 2–4 are used as the reference for the title's question and for the gap numbers in §5.1 (2.7%, 1.21%, 3.15%), but the manuscript never states how these Human rows were produced, what data split they used, which model architecture and hyperparameters were used, or what source they come from. Section 4.3 describes only the protocol for fine-tuning RoBERTa on LLM-annotated data; it does not describe a Human row protocol. If the Human rows are RoBERTa trained on the full training set Dtrain while the LLM rows are trained on the 30% subset T, then the reported gaps conflate annotation source with training-set size and cannot answer the paper's title question. Please document the human baseline protocol precisely, or remove/reframe all human-gap claims.
  2. [§3.1, §4.2] The paper says experiments are conducted 'on a 30% sample of the training set Dtrain' but does not specify how the 30% sample is drawn, whether this sample is the annotation target T or the context pool X, or how the random split is seeded. Section 3.1 defines a split of Dtrain into disjoint X and T, but §4.2 does not state the sizes of X and T or which of them corresponds to the 30% figure. Appendix D introduces 10% and 20% 'sample space' choices without defining the relationship to X or T. This underspecification matters because the amount of context available for retrieval and the amount of training data used to fine-tune RoBERTa are both confounded with the annotation method. Please give exact definitions of all splits and sizes.
  3. [§7, Tables 3–4] The conclusion in Section 7 that 'RAG-based methods consistently outperform both ICL and the baseline across all datasets' is too strong given the paper's own numbers. For example, in Table 3, Qwen2.5-72B on WNUT-17 with 25 context examples has ICL F1 = 51.49 but RAG w/OpenAI F1 = 49.05 and RAG w/ST F1 = 46.06; in Table 4, Qwen2.5-7B on WNUT-17 with 50 context examples has ICL F1 = 48.54 while RAG w/OpenAI has F1 = 44.75. Several other cells show ICL matching or exceeding RAG. The paper should replace the universal claim with a claim restricted to best configurations per dataset, or perform per-dataset, per-context-size statistical comparisons and report which differences are significant.
minor comments (4)
  1. [§5.1] The text says the best CoNLL-2003 result is 'just 2.7% below human-level annotation', but Table 2 gives Human F1 = 92.12 and the best model F1 = 89.72, an absolute difference of 2.40; if the intended comparison is relative, this should be stated explicitly.
  2. [Appendix E] The Friedman test results report a p-value of '7.71−18', which should be written as 7.71e−18 or as an actual p-value, and the report should state the number of datasets N and the number of models K used in the test.
  3. [§3.2.1] The sentence 'Dtrain is is split into X and T' contains a duplicated 'is' and should be corrected.
  4. [Figure 2] The x-axis labels '25 Ex. 50 Ex. 75 Ex.75 Ex.50 Ex.25 Ex.' are confusing and should be replaced with clear per-method grouping labels so the reader can map cells to context sizes and methods.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular dependency found: the paper reports direct benchmark measurements on held-out test sets, with no fitted parameter renamed as a prediction and no load-bearing self-citation chain.

full rationale

This paper is an empirical study, not a derivation. The central quantities are F1 scores of RoBERTa models fine-tuned on LLM-annotated subsets and evaluated on held-out test sets (Section 4.3), so the reported numbers are measurements rather than consequences of the method's own definitions. The RAG context-retrieval procedure (Section 3.2.4) uses only training-set examples X as a retrieval pool and annotates the disjoint subset T, so the annotation target is not itself used to select its own context. No parameter is fitted to the test set and then reported as a prediction. The paper contains self-citations (e.g., Ul Haq et al. 2024 is cited for token-label misalignment in Section 3.2.5), but this citation is background motivation for using structured output and does not carry the paper's comparative claims. The undocumented Human baseline rows in Tables 2-4, and the tension between the Section 7 claim that RAG 'consistently outperform[s] both ICL and the baseline across all datasets' and individual table cells where ICL outperforms RAG, are correctness and interpretability concerns, not circular reasoning. There is no equation in which an output variable is defined in terms of the claimed prediction, no fitted constant masquerading as a finding, and no uniqueness or external theorem imported from the authors' prior work. The evaluation is self-contained with respect to external benchmarks, and honest non-finding is therefore appropriate.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

No fitted constants, no invented entities. The paper is an empirical benchmark; its load-bearing assumptions are about the evaluation protocol, the comparability of the human baseline, and the parsability of LLM outputs.

assumptions (4)
  • domain assumption Fine-tuning RoBERTa on LLM-annotated training data is a valid proxy for annotation quality.
    Section 4.3: 'To assess the quality of annotations generated by LLMs, the RoBERTa model is fine-tuned on LLM-annotated datasets.' If LLM annotation errors are systematic, F1 may not reflect annotation quality.
  • domain assumption The 'Human' F1 scores in Tables 2-4 are comparable external references.
    Used to compute 'gap to human'. No citation or protocol is provided, so comparability is assumed.
  • domain assumption Sentence-level embedding similarity retrieves context examples that improve token-level NER annotation.
    Underlies the RAG method in Section 3.2.4; the paper tests it, but the claimed improvement depends on retrieval relevance.
  • domain assumption LLM outputs can be parsed into token-label annotations despite known structural inconsistencies.
    Section 3.2.5 notes that open models exhibit output inconsistencies; the paper does not state how malformed outputs were handled before fine-tuning.

how reviews work

0 comments
Cite this review

Pith. "Pith review of LLMs as Data Annotators: How Close Are We to Human Performance." pith.science (2026). https://pith.science/paper/ZP2YH5SC

@misc{pith2026250415022,
  author       = {Pith},
  title        = {Pith review of: LLMs as Data Annotators: How Close Are We to Human Performance},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ZP2YH5SC}},
  note         = {Machine review of arXiv:2504.15022}
}
abstract

In NLP, fine-tuning LLMs is effective for various applications but requires high-quality annotated data. However, manual annotation of data is labor-intensive, time-consuming, and costly. Therefore, LLMs are increasingly used to automate the process, often employing in-context learning (ICL) in which some examples related to the task are given in the prompt for better performance. However, manually selecting context examples can lead to inefficiencies and suboptimal model performance. This paper presents comprehensive experiments comparing several LLMs, considering different embedding models, across various datasets for the Named Entity Recognition (NER) task. The evaluation encompasses models with approximately $7$B and $70$B parameters, including both proprietary and non-proprietary models. Furthermore, leveraging the success of Retrieval-Augmented Generation (RAG), it also considers a method that addresses the limitations of ICL by automatically retrieving contextual examples, thereby enhancing performance. The results highlight the importance of selecting the appropriate LLM and embedding model, understanding the trade-offs between LLM sizes and desired performance, and the necessity to direct research efforts towards more challenging datasets.

Figures

Figures reproduced from arXiv: 2504.15022 by the authors.

Figure 1
Figure 1. Workflow of the proposed approach. Dtrain denotes the training data, X denotes the few human annotated examples, whereas T denotes the training instances to be annotated by LLM. For each entry Ti ∈ T , we extract M context examples from a vector store using a retriever module. Then, given an input sentence, the final prompt to LLM consists of the task description, the context examples in M, and input sentence. forma… view at source ↗
Figure 2
Figure 2. Heatmaps of the F1 scores across four datasets. The color scale represents performance, with red indicating higher scores reaching human-level, and blue indicating lower scores starting from the lowest performing model achieves an F1 score of 44.48, outperforming Llama3.1-8B, which scores 43.91. However, both models show a notable performance drop compared to their larger counterparts, suggesting that smaller models… view at source ↗
Figure 3
Figure 3. F1 scores for different context sizes (25, 50, and 75) and sample spaces (10% and 20%) for the RAG and ICL approach on the SKILLSPAN dataset, using the gpt-4o-mini model. The plot indicates that with a smaller sample size, the RAG approach performs comparably to ICL. insights. Across all datasets, RAG-based approaches improve annotation quality, with gpt-4o-mini and OpenAI embeddings achieving the best results. In c… view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Critical Difference diagram of average score ranks. The models connected with horizontal line shows no [PITH_FULL_IMAGE:figures/full_fig_p018_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

42 extracted references · 7 canonical work pages

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Sergei Bogdanov, Alexandre Constantin, Timothée Bernard, Benoit Crabbé, and Etienne Bernard. 2024. https://arxiv.org/abs/2402.15343 Nuner: Entity recognition encoder pre-training via llm-annotated data . Preprint, arXiv:2402.15343

    arXiv 2024

  4. [4]

    Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin...

    arXiv 2020

  5. [5]

    William Jay Conover. 1999. Practical Nonparametric Statistics, volume 350. John Wiley & Sons

    work page 1999

  6. [6]

    Leon Derczynski, Eric Nichols, Marieke van Erp, and Nut Limsopatham. 2017. https://doi.org/10.18653/v1/W17-4418 Results of the WNUT 2017 shared task on novel and emerging entity recognition . In Proceedings of the 3rd Workshop on Noisy User-generated Text, pages 140--147, Copenhagen, Denmark. Association for Computational Linguistics

    doi:10.18653/v1/w17-4418 2017

  7. [7]

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. ArXiv, abs/1810.04805

    arXiv 2019

  8. [8]

    Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Jingyuan Ma, Rui Li, Heming Xia, Jingjing Xu, Zhiyong Wu, Baobao Chang, Xu Sun, Lei Li, and Zhifang Sui. 2024. https://arxiv.org/abs/2301.00234 A survey on in-context learning . Preprint, arXiv:2301.00234

    arXiv 2024

Show all 42 references
  1. [9]

    Matthijs Douze, Alexandr Guzhva, Chengqi Deng, Jeff Johnson, Gergely Szilvasy, Pierre-Emmanuel Mazaré, Maria Lomeli, Lucas Hosseini, and Hervé Jégou. 2024. https://arxiv.org/abs/2401.08281 The faiss library

    2024 arXiv

  2. [10]

    Federico Errica, Giuseppe Siracusano, Davide Sanvito, and Roberto Bifulco. 2024. https://arxiv.org/abs/2406.12334 What did i do wrong? quantifying llms' sensitivity and consistency to prompt engineering . Preprint, arXiv:2406.12334

    2024 arXiv

  3. [11]

    Feng, Varun Gangal, Jason Wei, Sarath Chandar, Soroush Vosoughi, Teruko Mitamura, and Eduard Hovy

    Steven Y. Feng, Varun Gangal, Jason Wei, Sarath Chandar, Soroush Vosoughi, Teruko Mitamura, and Eduard Hovy. 2021. https://doi.org/10.18653/v1/2021.findings-acl.84 A survey of data augmentation approaches for NLP . In Findings of the Association for Computational Linguistics: ...

    2021 doi

  4. [12]

    Akshay Goel, Almog Gueta, Omry Gilon, Chang Liu, Sofia Erell, Lan Huong Nguyen, Xiaohong Hao, Bolous Jaber, Shashir Reddy, Rupesh Kartha, Jean Steiner, Itay Laish, and Amir Feder. 2023. https://arxiv.org/abs/2312.02296 Llms accelerate annotation for medical information extract...

    2023 arXiv

  5. [13]

    Xingwei He, Zhenghao Lin, Yeyun Gong, A-Long Jin, Hang Zhang, Chen Lin, Jian Jiao, Siu Ming Yiu, Nan Duan, and Weizhu Chen. 2024 a . https://arxiv.org/abs/2303.16854 Annollm: Making large language models to be better crowdsourced annotators . Preprint, arXiv:2303.16854

    2024 arXiv

  6. [14]

    Xingwei He, Zhenghao Lin, Yeyun Gong, A-Long Jin, Hang Zhang, Chen Lin, Jian Jiao, Siu Ming Yiu, Nan Duan, and Weizhu Chen. 2024 b . https://doi.org/10.18653/v1/2024.naacl-industry.15 A nno LLM : Making large language models to be better crowdsourced annotators . In Proceeding...

    2024 doi

  7. [15]

    Hugging Face . 2023. Transformers APIs . https://huggingface.co/docs/transformers/index. Accessed: 2023-01-21

    2023

  8. [16]

    Benoit Jacob, Skirmantas Kligys, Bo Chen, Menglong Zhu, Matthew Tang, Andrew Howard, Hartwig Adam, and Dmitry Kalenichenko. 2018. https://doi.org/10.1109/CVPR.2018.00286 Quantization and training of neural networks for efficient integer-arithmetic-only inference . In 2018 IEEE...

    2018

  9. [17]

    Mert Karabacak and Konstantinos Margetis. 2023. https://doi.org/10.7759/cureus.39305 Embracing large language models for medical applications: Opportunities and challenges . Cureus, 15(5):e39305

    2023 doi

  10. [18]

    u ttler, Mike Lewis, Wen-tau Yih, Tim Rockt\

    Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich K\" u ttler, Mike Lewis, Wen-tau Yih, Tim Rockt\" a schel, Sebastian Riedel, and Douwe Kiela. 2020. https://proceedings.neurips.cc/paper_files/paper/2020/file/6b493230205f78...

    2020

  11. [19]

    Yinghao Li, Rampi Ramprasad, and Chao Zhang. 2024. https://arxiv.org/abs/2402.13364 A simple but effective approach to improve structured language model output for information extraction . Preprint, arXiv:2402.13364

    2024 arXiv

  12. [20]

    Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. ArXiv, abs/1907.11692

    2019 arXiv

  13. [21]

    Yu Liu, Duantengchuan Li, Kaili Wang, Zhuoran Xiong, Fobo Shi, Jian Wang, Bing Li, and Bo Hang. 2024. https://doi.org/10.1016/j.ipm.2024.103809 Are llms good at structured outputs? a benchmark for evaluating structured output capabilities in llms . Information Processing & Man...

    2024

  14. [22]

    Yuji Naraki, Ryosuke Yamaki, Yoshikazu Ikeda, Takafumi Horie, and Hiroki Naganuma. 2024. https://arxiv.org/abs/2404.01334 Augmenting ner datasets with llms: Towards automated and refined annotation . Preprint, arXiv:2404.01334

    2024 arXiv

  15. [23]

    OpenAI. 2023. https://arxiv.org/abs/2303.08774 Gpt-4 technical report . Preprint, arXiv:2303.08774

    2023 arXiv

  16. [24]

    Pereira, Anabela Afonso, and Fátima Medeiros

    D. Pereira, Anabela Afonso, and Fátima Medeiros. 2015. https://doi.org/10.1080/03610918.2014.931971 Overview of friedman’s test and post-hoc analysis . Communications in Statistics - Simulation and Computation, 44:2636--2653

    2015

  17. [25]

    Nils Reimers and Iryna Gurevych. 2019. https://arxiv.org/abs/1908.10084 Sentence-bert: Sentence embeddings using siamese bert-networks . Preprint, arXiv:1908.10084

    2019 arXiv

  18. [26]

    Stefan Strohmeier. 2022. Handbook of Research on Artificial Intelligence in Human Resource Management. Edward Elgar Publishing

    2022

  19. [27]

    Zhen Tan, Dawei Li, Alimohammad Beigi, Song Wang, Ruocheng Guo, Amrita Bhattacharjee, Bohan Jiang, Mansooreh Karami, Jundong Li, Lu Cheng, and Huan Liu. 2024. https://api.semanticscholar.org/CorpusID:267770019 Large language models for data annotation: A survey . ArXiv, abs/2402.13446

    2024 arXiv

  20. [28]

    Gemini Team. 2024 a . https://arxiv.org/abs/2312.11805 Gemini: A family of highly capable multimodal models . Preprint, arXiv:2312.11805

    2024 arXiv

  21. [29]

    Qwen Team. 2024 b . https://qwenlm.github.io/blog/qwen2.5/ Qwen2.5: A party of foundation models

    2024

  22. [30]

    Maksim Terpilowski. 2019. https://doi.org/10.21105/joss.01169 scikit-posthocs: Pairwise multiple comparison tests in python . The Journal of Open Source Software, 4(36):1169

    2019 doi

  23. [31]

    Tjong Kim Sang and Fien De Meulder

    Erik F. Tjong Kim Sang and Fien De Meulder. 2003. Introduction to the C o NLL -2003 shared task: Language-independent named entity recognition. In Proceedings of the Seventh Conference on Natural Language Learning at HLT - NAACL 2003 , pages 142--147

    2003

  24. [32]

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. https://arxiv.org/abs/2302.13971 Llama:...

    2023 arXiv

  25. [33]

    Muhammad Uzair Ul Haq, Paolo Frazzetto, Alessandro Sperduti, and Giovanni Da San Martino. 2024. https://doi.org/10.1145/3605098.3636010 Improving soft skill extraction via data augmentation and embedding manipulation . In Proceedings of the 39th ACM/SIGAPP Symposium on Applied...

    2024

  26. [34]

    Ashok Urlana, Charaka Vinayak Kumar, Ajeet Kumar Singh, Bala Mallikarjunarao Garlapati, Srinivasa Rao Chalamala, and Rahul Mishra. 2024. https://arxiv.org/abs/2402.14558 Llms with industrial lens: Deciphering the challenges and prospects -- a survey . Preprint, arXiv:2402.14558

    2024 arXiv

  27. [35]

    Shuhe Wang, Xiaofei Sun, Xiaoya Li, Rongbin Ouyang, Fei Wu, Tianwei Zhang, Jiwei Li, and Guoyin Wang. 2023. https://arxiv.org/abs/2304.10428 Gpt-ner: Named entity recognition via large language models . Preprint, arXiv:2304.10428

    2023 arXiv

  28. [36]

    Shuohang Wang, Yang Liu, Yichong Xu, Chenguang Zhu, and Michael Zeng. 2021. https://doi.org/10.18653/v1/2021.findings-emnlp.354 Want to reduce labeling cost? GPT -3 can help . In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 4195--4205, Punta Can...

    2021 doi

  29. [37]

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. 2023. https://arxiv.org/abs/2201.11903 Chain-of-thought prompting elicits reasoning in large language models . Preprint, arXiv:2201.11903

    2023 arXiv

  30. [38]

    Jia-Yu Yao, Kun-Peng Ning, Zhen-Hui Liu, Mu-Nan Ning, Yu-Yang Liu, and Li Yuan. 2024. https://arxiv.org/abs/2310.01469 Llm lies: Hallucinations are not bugs, but features as adversarial examples . Preprint, arXiv:2310.01469

    2024 arXiv

  31. [39]

    Amir Zeldes. 2017. https://doi.org/http://dx.doi.org/10.1007/s10579-016-9343-x The GUM corpus: Creating multilayer resources in the classroom . Language Resources and Evaluation, 51(3):581--612

    2017 doi

  32. [40]

    Mike Zhang, Kristian N rgaard Jensen, Sif Dam Sonniks, and Barbara Plank. 2022 a . Skillspan: Hard and soft skill extraction from english job postings. In North American Chapter of the Association for Computational Linguistics

    2022

  33. [41]

    Zhuosheng Zhang, Aston Zhang, Mu Li, and Alex Smola. 2022 b . https://arxiv.org/abs/2210.03493 Automatic chain of thought prompting in large language models . Preprint, arXiv:2210.03493

    2022 arXiv

  34. [42]

    Ran Zhou, Xin Li, Ruidan He, Lidong Bing, Erik Cambria, Luo Si, and Chunyan Miao. 2022. https://doi.org/10.18653/v1/2022.acl-long.160 MELM : Data augmentation with masked entity language modeling for low-resource NER . In Proceedings of the 60th Annual Meeting of the Associati...

    2022 doi

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.