Pith. sign in

REVIEW 4 major objections 4 minor 50 references

A Benchmark and Robustness Study of In-Context-Learning with Large Language Models in Music Entity Detection

T0 review · 4 major / 4 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read In-context-learning LLMs outperform fine-tuned small models on music entity detection, but the margin collapses for works the model did not memorize during pre-training.

desk verdict Solid new dataset and benchmark for music NER with LLMs; the exposure conclusion rests on a questionable proxy and needs tightening, but the core result holds. read the letter →

arxiv 2412.11851 v1 pith:XJSGKNVI submitted 2024-12-16 cs.CL cs.MM

Simon Hachmeier , Robert Jäschke This is my paper · ORCID
classification cs.CLcs.MM
keywords namedentityrecognitionmusicdetectionin-contextlearninglargelanguagemodelsexposurefactualmemorizationuser-generatedcontentrobustness
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper benchmarks four large language models against fine-tuned BERT and RoBERTa on a new music-entity dataset built from YouTube video titles and Reddit music-request posts, and then tests robustness to unseen and perturbed entity strings. The benchmark shows that LLMs used with in-context-learning reach higher macro F1 than the SLM baselines, with GPT-4o-mini the best at 0.84 with 35 few-shot examples. The second experiment reveals a catch: recall on works of art that a model fails on a two-question factual memorization test drops by up to 0.24, and controlled synthesis with never-before-seen post-cutoff entities produces the largest error rates. The paper concludes that part of the LLMs' apparent superiority is due to pre-training exposure to the exact musical works, not to a genuinely stronger ability to recognize novel entity strings.

What carries the argument

The paper's load-bearing machinery is a two-question Factual Memorization Test (FMT) built from the SHS100K cover-song knowledge base: for each musical work the model is asked who originally performed it and who wrote it, and the two answers are scored as Passed, Partial, or Failed. A second component is cloze-based data synthesis: 1,067 unique contexts are extracted from the annotated datasets, entities from three exposure groups (FMT Passed, FMT Failed, and post-cutoff debut artists from MusicBrainz) are inserted, and two levels of character/word perturbations and abbreviations are applied. The FMT is the proxy that ties NER recall back to pre-training exposure, while the cloze synthesis isolates context-driven detection from entity-driven recognition.

What would settle it

Compare WoA recall across two groups of works that both pass the FMT but whose exact title strings have very different frequencies in typical training text (e.g., frequently covered singles versus obscure album tracks); if recall is identical, the effect is about the work rather than the string, and if recall tracks string frequency, the FMT is not the right exposure proxy.

Watch

Extended reading notes

Core claim

The central claim is that LLMs with in-context-learning outperform fine-tuned SLMs on music entity detection in user-generated content, but that a substantial part of this advantage is an artifact of entity exposure: the better the model remembers the musical work from pre-training, the higher its recall on that work's title. The paper demonstrates this by splitting benchmark results according to a factual memorization test and by synthesizing three matched datasets whose entities are either known, partially known, or released after the model's knowledge cutoff. In those matched settings, error rates rise most sharply for the never-seen entities, and the effect of exposure is larger than the effect of realistic typos and abbreviations. Thus the authors argue that LLM-based music NER in the wild will inherit the same exposure bias long documented for smaller language models.

Load-bearing premise

The central argument depends on the assumption that a two-question factual memorization test (who originally performed the song, who wrote it) is a valid measure of whether the model has memorized the exact entity string that appears in the NER task; if the test can be passed through reasoning or exposure to related entities, the measured recall drop cannot be attributed to pre-training exposure of that specific string.

Editorial extensions

If this is right

  • Practitioners using LLM-based music NER on long-tail catalogs should expect materially lower recall than benchmark numbers on popular hits suggest, because recall drops by up to 0.24 for works the model cannot recall.
  • Few-shot in-context-learning helps across all tested LLMs, so retrieval-based example selection (tf-idf) is preferable to zero-shot prompting for music entity detection.
  • Data augmentation with typos and abbreviations will not close the gap on genuinely unseen entities, since exposure has a stronger effect than perturbation; gazetteers or retrieval augmentation would be more promising.
  • Future NER benchmark comparisons should control for pre-training exposure, since the exposure effect documented for SLMs persists in LLMs and affects the ranking of models.
  • The released MusicUGC-NER dataset provides a shared, human-annotated IOB testbed that joins YouTube metadata and Reddit conversation, enabling direct comparison of music-domain NER in noisy user-generated text.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Replacing the FMT proxy with a direct string-level memorization probe (e.g., asking the model to complete the exact utterance) would sharpen the causal link the paper leaves implicit, separating 'knows the work' from 'has seen this exact string'.
  • The post-cutoff group gives a cleaner estimate of intrinsic NER ability than any in-distribution split, so reporting error rates on never-seen entities could become a standardized robustness score for future LLMs.
  • The same cloze-and-perturbation protocol transfers to other creative-content domains such as movies, books, or games, where entity names are non-regular and user text is noisy; the FMT would simply be replaced by a domain-appropriate factual probe.
  • Because the exposure effect outweighs context, adding an explicit 'unknown entity' option to the prompt might reduce spurious and incorrect labels on unseen works, a testable design change directly motivated by the paper's error analysis.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper introduces MusicUGC-NER, a human-annotated dataset for detecting music entities (artists and works of art) in user-generated YouTube video titles, and combines it with the existing MusicRecoNER Reddit dataset. It benchmarks four LLMs (FireFunction-v2, GPT-4o-mini, Llama3.1-70B, Mixtral-8x22B) under in-context learning with tf-idf-based example retrieval against fine-tuned BERT and RoBERTa baselines, reporting strict F1 scores under five-fold cross-validation. In a second robustness study, the authors use GPT-4o-mini on a synthesized cloze dataset with entities from three groups (FMT-passed, FMT-failed, and 2024 post-cutoff debut artists from MusicBrainz) and two perturbation levels, analyzing error types. The paper claims that LLMs outperform SLMs in this ICL setting and that pre-training entity exposure has a large impact on detection performance, particularly for works-of-art.

Significance. The dataset and benchmark address an applied gap in music-domain NER over noisy user-generated content, and the authors should be credited for releasing the data, reporting high inter-annotator agreement (Kappa 0.93), and for the post-cutoff control with 2024 debut artists, which is a genuinely stronger unseen-entity manipulation than the FMT-based grouping. The perturbation taxonomy and the explicit matching-precision/recall evaluation are useful contributions. However, the FMT proxy for entity-string exposure is not validated, and the benchmark/robustness claims lack uncertainty quantification; these issues currently weaken the central causal conclusion. If the authors address these points, the paper would be a solid reference for practitioners choosing between fine-tuned SLMs and LLM-based ICL for music entity detection.

major comments (4)
  1. [Section 4.1, Table 4] The FMT is used as the measure of pre-training entity exposure, but it tests canonical work–attribute knowledge (e.g., 'Who originally performed Yesterday?'), not the exact surface strings that appear in the D-YT video titles. A model can answer FMT questions through general world knowledge without having memorized abbreviated or misspelled YouTube-style utterances, and it can fail for reasons unrelated to the surface form. Therefore the .24 recall gap between Passed and Failed FMT groups in Table 4 may be confounded by entity frequency, length, or ambiguity, and the causal claim that exposure to the exact entity strings drives the drop is not established. The post-cutoff control in Section 4.2 is cleaner; the FMT-based interpretation should be reframed or supplemented with a string-level memorization probe (e.g., next-token prediction on the actual utterances).
  2. [Section 6.1, Tables 3 and 4] Table 3 reports only mean F1 without standard deviations or significance tests. Differences such as GPT-4o-mini .84 vs Llama3.1-70B .82 are likely within fold-to-fold variation, so the claim that LLMs outperform SLMs is not yet quantified with confidence intervals. The FMT subsets in Table 4 have small, unbalanced supports (e.g., n=103 for GPT-4o-mini Failed vs n=229 Passed), making the .24 recall gap sensitive to a few instances. I request per-fold results, bootstrap intervals, or paired significance tests (e.g., McNemar) for the main comparisons.
  3. [Section 6.2, Figures 3 and 4] The robustness analysis plots error proportions without uncertainty bounds, and it is unclear whether the reported differences (e.g., the larger Post-Cutoff errors in Figure 3, or the perturbation-level trends in Figure 4) are reproducible or reflect sampling noise. Additionally, the six error categories are not independent: a misclassification contributes to both a missed and an incorrect count for the two classes, so the statement that 'the effect of exposure appears to be stronger than the effect of perturbation' is not supported by a statistical comparison. Please report error bars and a per-cloze paired comparison, or a single aggregate error measure.
  4. [Section 5.2, Figure 8] The ICL prompt is taken from a previous study on conversational Reddit queries and instructs the model to extract entities from 'a user request for music suggestions'; however, half of the benchmark data (D-YT) consists of YouTube video titles, which are not such requests. Using this prompt unmodified for YouTube titles may systematically disadvantage the LLMs relative to fine-tuned SLMs, which see task-native training data. The authors should either adapt the instruction to the YouTube setting or include a task-matched prompt control to rule out this confound.
minor comments (4)
  1. [Table 3] Table 3, Mixtral-8x22B, k=0, D-RD+YT: the reported macro average .80 is inconsistent with the Artist .73 and WoA .67, whose mean is .70; this appears to be an arithmetic error.
  2. [Table 4 caption] Table 4 caption: 'per outcome of the of the FMT' should be 'per outcome of the FMT'.
  3. [Section 4.2] The perturbation generation uses p=0.5 but the exact distributions for character/word deletion, insertion, substitution and the interaction between Level-1 and Level-2 are only specified in the repository; for reproducibility, the paper should state these distributions, the random seed, and explicitly say whether abbreviations can be applied to WoA tokens.
  4. [Section 4.2] The term 'cloze' is used for masked templates that are filled with other entities, rather than for an infilling task; a brief clarification of this usage would avoid confusion.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: benchmark and robustness results are empirical measurements against external corpora and controls; the FMT exposure proxy is a modeling assumption, not a fitted input to the reported predictions.

full rationale

The paper's central claims are (i) LLMs with ICL outperform SLMs on MusicUGC-NER and (ii) entity exposure affects NER robustness. Claim (i) is a direct empirical measurement on the newly annotated D-YT and the joint D-RD+YT dataset, using five-fold cross-validation and standard F1; no equation in the paper defines the reported scores in terms of a fitted parameter or an assumed outcome. Claim (ii) is operationalized in two ways: a factual memorization test (Section 4.1) and a post-cutoff/MusicBrainz unseen-entity control (Sections 4.1-4.2). The FMT is a proxy for pre-training exposure, not a definition: the paper states it "model[s] factual memorization" and then compares NER recall across FMT outcomes (Table 4, Figure 3). Whether the FMT validly measures exposure is a correctness/validity concern, not a circularity concern, because the NER results are measured, not derived from the FMT labels. The post-cutoff control provides an independent manipulation. The paper cites the authors' prior work (Hachmeier and Jäschke, 2024) for the ICL prompt and tf-idf few-shot sampling, but this is a methodological self-citation and is not load-bearing: the current study independently re-evaluates random vs tf-idf sampling (Appendix A.5, Figure 9), and the benchmark conclusions rest on the current measurements rather than on the prior paper's results. No prediction in the paper reduces by construction to an input, and no equation equates a fitted quantity with a reported outcome. Thus there is no significant circularity.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

This is an empirical study with no mathematical derivation. The central claims rest on dataset construction choices and on two proxy assumptions: FMT scores for exposure and MusicBrainz debuts for unseen entities. No new physical or theoretical entities are introduced.

free parameters (3)
  • perturbation_probability = 0.5
    Hand-chosen probability for applying perturbations in the robustness study; the comparison between exposure and perturbation effects depends on this value (Section 4.2).
  • few_shot_size_k = 35
    Chosen as the best performing setting for GPT-4o-mini and used for all reported robustness results (Section 6.1); F1 varies with k.
  • fmt_outcome_classification = Passed/Partial/Failed
    Hand-defined thresholds over two factual questions determine exposure groups; small support in the Failed group influences Table 4 (Section 4.1).
assumptions (4)
  • domain assumption FMT performance is a valid proxy for pre-training entity exposure
    The robustness study treats two correct factual answers about a work as evidence that the model memorized the work, and links this to NER recall (Section 4.1).
  • domain assumption Post-Cutoff entities are unseen by GPT-4o-mini
    2024 debut artists from MusicBrainz are assumed to be after the model's knowledge cutoff, making them unseen entities (Section 4.1).
  • domain assumption External metadata (SHS100K, MusicBrainz) is accurate
    SHS100K provides the ground truth for FMT and entity variations; MusicBrainz provides post-cutoff entities (Sections 3.1, 4.1).
  • domain assumption Temperature 0 yields deterministic LLM outputs
    All LLM experiments set temperature to 0 and report single runs without seeds or variance (Section 5.2).

how reviews work

0 comments
Cite this review

Pith. "Pith review of A Benchmark and Robustness Study of In-Context-Learning with Large Language Models in Music Entity Detection." pith.science (2026). https://pith.science/paper/XJSGKNVI

@misc{pith2026241211851,
  author       = {Pith},
  title        = {Pith review of: A Benchmark and Robustness Study of In-Context-Learning with Large Language Models in Music Entity Detection},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XJSGKNVI}},
  note         = {Machine review of arXiv:2412.11851}
}
read the original abstract

Detecting music entities such as song titles or artist names is a useful application to help use cases like processing music search queries or analyzing music consumption on the web. Recent approaches incorporate smaller language models (SLMs) like BERT and achieve high results. However, further research indicates a high influence of entity exposure during pre-training on the performance of the models. With the advent of large language models (LLMs), these outperform SLMs in a variety of downstream tasks. However, researchers are still divided if this is applicable to tasks like entity detection in texts due to issues like hallucination. In this paper, we provide a novel dataset of user-generated metadata and conduct a benchmark and a robustness study using recent LLMs with in-context-learning (ICL). Our results indicate that LLMs in the ICL setting yield higher performance than SLMs. We further uncover the large impact of entity exposure on the best performing LLM in our study.

Figures

Figures reproduced from arXiv: 2412.11851 by the authors.

Figure 1
Figure 1. Relative positions of the utterances per class [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Questions of our factual memorization test [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 4
Figure 4. Error proportions per metric per imposed [PITH_FULL_IMAGE:figures/full_fig_p008_4.png] view at source ↗
Figures from the paper (5 more)
Figure 5
Figure 5. Figure 5: Cumulative distribution functions of F1 scores [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: Graphical user interface of our annotation [PITH_FULL_IMAGE:figures/full_fig_p013_6.png]
Figure 7
Figure 7. Figure 7: Fractions of correctly answered questions in [PITH_FULL_IMAGE:figures/full_fig_p014_7.png]
Figure 9
Figure 9. Figure 9: Performance on D-RD+YT using random sampling as opposed to tf-idf-sampling. performance [PITH_FULL_IMAGE:figures/full_fig_p014_9.png]
Figure 11
Figure 11. Figure 11: Clozes after dimensionality reduction with [PITH_FULL_IMAGE:figures/full_fig_p015_11.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

50 extracted references · 26 canonical work pages

  1. [1]

    Dhananjay Ashok and Zachary C Lipton. 2023. Promptner: Prompting for named entity recognition. arXiv preprint arXiv:2305.15444

    arXiv 2023

  2. [2]

    David S. Batista. 2018. https://www.davidsbatista.net/blog/2018/05/09/Named_Entity_Evaluation/ Named-entity evaluation metrics based on entity-level . Accessed: 2024-09-13

    work page 2018

  3. [3]

    Adrian MP Brasoveanu, Albert Weichselbraun, and Lyndon Nixon. 2020. In media res: a corpus for evaluating named entity linking with creative works. In Proceedings of the 24th Conference on Computational Natural Language Learning, pages 355--364

    work page 2020

  4. [4]

    Samuel Colvin, Eric Jolibois, Hasan Ramezani, Adrian Garcia Badaracco, Terrence Dorsey, David Montague, Serge Matveenko, Marcelo Trylesinski, Sydney Runkle, David Hewitt, and Alex Hall. 2023. https://docs.pydantic.dev/latest/ Pydantic . If you use this software, please cite it as below

    work page 2023

  5. [5]

    Leon Derczynski, Eric Nichols, Marieke van Erp, and Nut Limsopatham. 2017. https://doi.org/10.18653/v1/W17-4418 Results of the WNUT 2017 shared task on novel and emerging entity recognition . In Proceedings of the 3rd Workshop on Noisy User-generated Text, pages 140--147, Copenhagen, Denmark. Association for Computational Linguistics

    doi:10.18653/v1/w17-4418 2017

  6. [6]

    Jacob Devlin, Ming - Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. https://arxiv.org/abs/1810.04805 BERT: pre-training of deep bidirectional transformers for language understanding . CoRR, abs/1810.04805

    arXiv 2018

  7. [7]

    Zepeng Ding, Ruiyang Ke, Wenhao Huang, Guochao Jiang, Yanda Li, Deqing Yang, Yanghua Xiao, and Jiaqing Liang. 2024. Adaptive reinforcement learning planning: Harnessing large language models for complex information extraction. arXiv preprint arXiv:2406.11455

    work page Pith review arXiv 2024

  8. [8]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurelien Rodriguez, Austen Gregerson, Ava Spataru, Baptiste Roziere, Bethany...

    arXiv 2024

Show all 50 references
  1. [9]

    David Duki \'c and Jan S najder. 2024. Looking right is sometimes right: Investigating the capabilities of decoder-only llms for sequence labeling. In Findings of the Association for Computational Linguistics ACL 2024, pages 14168--14181

    2024

  2. [10]

    Elena Epure and Romain Hennequin. 2023. https://doi.org/10.18653/v1/2023.eacl-main.92 A human subject study of named entity recognition in conversational music recommendation queries . In Proceedings of the 17th Conference of the European Chapter of the Association for Computa...

    2023 doi

  3. [11]

    Epure and Romain Hennequin

    Elena V. Epure and Romain Hennequin. 2022. https://aclanthology.org/2022.lrec-1.151 Probing pre-trained auto-regressive language models for named entity typing and recognition . In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 1408--1417, Ma...

    2022

  4. [12]

    Xiaoning Feng, Xiaohong Han, Simin Chen, and Wei Yang. 2024. https://doi.org/10.1145/3664812 Llmeffichecker: Understanding and testing efficiency degradation of large language models . ACM Trans. Softw. Eng. Methodol., 33(7)

    2024 doi

  5. [13]

    Pawel Garbacki and Benny Chen. 2024. https://fireworks.ai/blog/firefunction-v2-launch-post Firefunction-v2: Function calling capability on par with GPT4o at 2.5x the speed and 10\ Accessed: 2024-09-06

    2024

  6. [14]

    Simon Hachmeier and Robert Jäschke. 2024. Information extraction of music entities in conversational music queries. In Proceedings of the 3rd Workshop on NLP for Music and Audio (NLP4MusA)

    2024

  7. [15]

    Valentin Hartmann, Anshuman Suri, Vincent Bindschaedler, David Evans, Shruti Tople, and Robert West. 2023. Sok: Memorization in general-purpose large language models. arXiv preprint arXiv:2310.18362

    2023 arXiv

  8. [16]

    Wenhao Huang, Jiaqing Liang, Zhixu Li, Yanghua Xiao, and Chuanjun Ji. 2023. https://doi.org/10.18653/v1/2023.findings-acl.863 Adaptive ordered information extraction with deep reinforcement learning . In Findings of the Association for Computational Linguistics: ACL 2023, page...

    2023 doi

  9. [17]

    Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie-Anne...

    2024 arXiv

  10. [18]

    Valentin Jijkoun, Mahboob Alam Khalid, Maarten Marx, and Maarten De Rijke. 2008. Named entity normalization in user generated content. In Proceedings of the second workshop on Analytics for noisy unstructured text data, pages 23--30

    2008

  11. [19]

    Sung Jae Jung, Hajung Kim, and Kyoung Sang Jang. 2024. Llm based biological named entity recognition from scientific literature. In 2024 IEEE International Conference on Big Data and Smart Computing (BigComp), pages 433--435. IEEE

    2024

  12. [20]

    Zongxi Li, Xianming Li, Yuzhang Liu, Haoran Xie, Jing Li, Fu-lee Wang, Qing Li, and Xiaoqin Zhong. 2023. Label supervised llama finetuning. arXiv preprint arXiv:2310.01208

    2023 arXiv

  13. [21]

    Sandra Liljeqvist. 2016. Named entity recognition for search queries in the music domain

    2016

  14. [22]

    Bill Yuchen Lin, Dong-Ho Lee, Ming Shen, Ryan Moreno, Xiao Huang, Prashant Shiralkar, and Xiang Ren. 2020 a . Triggerner: Learning with entity triggers as explanations for named entity recognition. arXiv preprint arXiv:2004.07493

    2020 arXiv

  15. [23]

    Hongyu Lin, Yaojie Lu, Jialong Tang, Xianpei Han, Le Sun, Zhicheng Wei, and Nicholas Jing Yuan. 2020 b . https://doi.org/10.18653/v1/2020.emnlp-main.592 A rigorous study on named entity recognition: Can fine-tuning pretrained model lead to the promised land? In Proceedings of ...

    2020 doi

  16. [24]

    Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. https://arxiv.org/abs/1907.11692 Roberta: A robustly optimized bert pretraining approach . Preprint, arXiv:1907.11692

    2019 arXiv

  17. [25]

    Zihan Liu, Yan Xu, Tiezheng Yu, Wenliang Dai, Ziwei Ji, Samuel Cahyawijaya, Andrea Madotto, and Pascale Fung. 2021. Crossner: Evaluating cross-domain named entity recognition. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 13452--13460

    2021

  18. [26]

    Liwen Ma and Weifeng Liu. 2021. An enhanced method for entity trigger named entity recognition based on pos tag embedding. In 2021 IEEE 7th International Conference on Cloud Computing and Intelligent Systems (CCIS), pages 89--93. IEEE

    2021

  19. [27]

    Yubo Ma, Yixin Cao, Yong Hong, and Aixin Sun. 2023. https://doi.org/10.18653/v1/2023.findings-emnlp.710 Large language model is not a good few-shot information extractor, but a good reranker for hard samples! In Findings of the Association for Computational Linguistics: EMNLP ...

    2023 doi

  20. [28]

    Zhiwei Ma, Javier E Santo, Greg Lackey, Hari Viswanathan, and Daniel O'Malley. 2024. Information extraction from historical well records using a large language model. arXiv preprint arXiv:2405.05438

    2024 arXiv

  21. [29]

    Mistral AI Team . 2024. https://mistral.ai/news/mixtral-8x22b/ Cheaper, better, faster, stronger continuing to push the frontier of ai and making it accessible to all. Accessed: 2024-09-06

    2024

  22. [30]

    OpenAI. 2024. https://openai.com/index/hello-gpt-4o/ Hello GPT-4o . Accessed: 2024-09-11

    2024

  23. [31]

    OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jeff ...

    2024 arXiv

  24. [32]

    Sergio Oramas, Luis Espinosa Anke, Mohamed Sordo, Horacio Saggion, and Xavier Serra. 2016. https://aclanthology.org/L16-1528 ELMD : An automatically generated entity linking gold standard dataset in the music domain . In Proceedings of the Tenth International Conference on Lan...

    2016

  25. [33]

    Letian Peng, Zilong Wang, Feng Yao, Zihan Wang, and Jingbo Shang. 2024. Metaie: Distilling a meta model from llm for all kinds of information extraction tasks. arXiv preprint arXiv:2404.00457

    2024 arXiv

  26. [34]

    Felipe Pezoa, Juan L Reutter, Fernando Suarez, Mart \' n Ugarte, and Domagoj Vrgo c . 2016. Foundations of json schema. In Proceedings of the 25th International Conference on World Wide Web, pages 263--273. International World Wide Web Conferences Steering Committee

    2016

  27. [35]

    Lorenzo Porcaro and Horacio Saggion. 2019. Recognizing musical entities in user-generated content. Computaci \'o n y Sistemas , 23(3):1079--1088

    2019

  28. [36]

    Ramshaw and Mitchell P

    Lance A. Ramshaw and Mitchell P. Marcus. 1995. https://arxiv.org/abs/cmp-lg/9505040 Text chunking using transformation-based learning . Preprint, arXiv:cmp-lg/9505040

    1995 arXiv

  29. [37]

    Ekagra Ranjan and Naman Poddar. 2022. https://arxiv.org/abs/2204.01848 Multilingual abusiveness identification on code-mixed social media text . Preprint, arXiv:2204.01848

    2022 arXiv

  30. [38]

    Isabel Segura-Bedmar, Paloma Mart \' nez, and Mar \' a Herrero-Zazo. 2013. https://aclanthology.org/S13-2056 S em E val-2013 task 9 : Extraction of drug-drug interactions from biomedical texts ( DDIE xtraction 2013) . In Second Joint Conference on Lexical and Computational Sem...

    2013

  31. [39]

    Karen Sparck Jones. 1972. A statistical interpretation of term specificity and its application in retrieval. Journal of documentation, 28(1):11--21

    1972

  32. [40]

    Xiaofei Sun, Linfeng Dong, Xiaoya Li, Zhen Wan, Shuhe Wang, Tianwei Zhang, Jiwei Li, Fei Cheng, Lingjuan Lyu, Fei Wu, and Guoyin Wang. 2023. https://arxiv.org/abs/2306.09719 Pushing the limits of chatgpt on nlp tasks . Preprint, arXiv:2306.09719

    2023 arXiv

  33. [41]

    Shuhe Wang, Xiaofei Sun, Xiaoya Li, Rongbin Ouyang, Fei Wu, Tianwei Zhang, Jiwei Li, and Guoyin Wang. 2023. https://arxiv.org/abs/2304.10428 Gpt-ner: Named entity recognition via large language models . Preprint, arXiv:2304.10428

    2023 arXiv

  34. [42]

    Wenjia Xu and Yangyang Qi. 2022. Gazetteer enhanced named entity recognition for musical user-generated content. In 2022 3rd International Conference on Computer Science and Management Technology (ICCSMT), pages 40--43. IEEE

    2022

  35. [43]

    Xiaoshuo Xu, Xiaoou Chen, and Deshun Yang. 2018. https://doi.org/10.1109/ICME.2018.8486531 Key-invariant convolutional neural network toward efficient cover song identification . In 2018 IEEE International Conference on Multimedia and Expo (ICME), pages 1--6

    2018

  36. [44]

    Junjie Ye, Nuo Xu, Yikun Wang, Jie Zhou, Qi Zhang, Tao Gui, and Xuanjing Huang. 2024. Llm-da: Data augmentation via large language models for few-shot named entity recognition. arXiv preprint arXiv:2402.14568

    2024 arXiv

  37. [45]

    Furkan Yesiler, Guillaume Doras, Rachel M Bittner, Christopher J Tralie, and Joan Serr \`a . 2021. Audio-based musical version identification: Elements and challenges. IEEE Signal Processing Magazine, 38(6):115--136

    2021

  38. [46]

    Mozhi Zhang, Hang Yan, Yaqian Zhou, and Xipeng Qiu. 2023. Promptner: A prompting method for few-shot named entity recognition via k nearest neighbor search. arXiv preprint arXiv:2305.12217

    2023 arXiv

  39. [47]

    Zhen Zhang, Yuhua Zhao, Hang Gao, and Mengting Hu. 2024. https://doi.org/10.1145/3589334.3645414 Linkner: Linking local named entity recognition models to large language models using uncertainty . In Proceedings of the ACM on Web Conference 2024, WWW '24, page 4047–4058, New Y...

    2024

  40. [48]

    Wenxuan Zhou, Sheng Zhang, Yu Gu, Muhao Chen, and Hoifung Poon. 2024. https://arxiv.org/abs/2308.03279 Universalner: Targeted distillation from large language models for open named entity recognition . Preprint, arXiv:2308.03279

    2024 arXiv

  41. [49]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  42. [50]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.