{"id":"c8615cbc-efc5-4d31-b3db-8fba42dc4d04","arxiv_id":"2412.11851","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Large language models with in-context learning outperform fine-tuned BERT and RoBERTa for music entity detection in user-generated content, but their edge shrinks for entities not memorized during pre-training.","lead":"This paper introduces a new dataset of music entities in YouTube video titles and Reddit posts, and benchmarks large language models against smaller fine-tuned models. It finds that in-context-learning with LLMs, especially GPT-4o-mini, outperforms the smaller models, but that performance drops sharply for music entities the model appears not to have memorized.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The factual memorization test (FMT) is used as a proxy for pre-training exposure, but it measures knowledge of original artist/composer from canonical titles, not the surface form of the exact UGC mention; the exposure conclusion may be confounded.","rationale":"The reader's weakest_assumption is exactly this FMT proxy problem: passing a two-question factual test does not guarantee memorization of the particular YouTube-style utterance. My analysis confirms and sharpens this by noting that the FMT uses canonical titles rather than surface strings, that 'Partial' scoring is loose, and that FMT groups may be confounded with entity frequency or length. This concern is load-bearing because the paper's headline claim about entity exposure's large impact depends on it. However, the benchmark claim (LLMs with ICL outperform SLMs) is supported by Table 3 even without the exposure analysis, and the post-cutoff synthetic experiment provides a more direct test of unseen entities, though it is not used to support the specific passed-vs-failed comparison. Thus the central benchmark result stands, while the exposure-specific conclusion remains conditional. The proposed concrete test would settle whether the FMT proxy holds; until then, the reader's CONDITIONAL verdict is appropriate and no change is needed.","tokens_in":16902,"tokens_out":4404,"duration_ms":40876,"concrete_test":"Re-run the FMT scoring using the exact UGC utterance (the surface string from the YouTube title, after the paper's own pre-processing) as the subject of the two factual questions, instead of the canonical SHS title. Recompute the FMT group assignments and the WoA recall per group in Table 4. If a non-trivial fraction (>10%) of entities change FMT outcome, or if the passed-vs-failed recall gap shrinks by more than half, the original FMT was not measuring exposure to the specific mention strings, and the exposure conclusion in Section 6.1 requires a different operationalization.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's second central claim—that pre-training entity exposure has a large impact on NER performance (Section 6.1, Table 4)—rests entirely on the FMT as a measure of exposure. The FMT (Section 4.1) asks 'Who originally performed?' and 'Who wrote?' using the canonical SHS title (e.g., 'Yesterday'), not the surface string that appears in the UGC (e.g., 'yesterday (acoustic)' or a misspelled variant). Passing the FMT therefore reflects general factual knowledge about a musical work, which can be acquired without memorizing the exact utterance strings used in the NER task. A model can answer 'The Beatles' for 'Yesterday' via common knowledge, yet still fail to recognize a typo-laden or abbreviated mention in a YouTube title. Conversely, failing a FMT question may be due to question format, ambiguity, or a related-entity match (the 'Partial' outcome includes any covering artist), not to the absence of the mention string in pre-training. The FMT groups in D-YT are small (e.g., 103 failed vs. 229 passed for GPT-4o-mini) and may differ systematically in entity frequency, length, or ambiguity, so the .24 recall drop between passed and failed FMT could be driven by these confounds. Because the post-cutoff control (Section 4.1) is a stronger unseen-entity manipulation, the specific causal conclusion about exposure from the FMT comparison is not yet established.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces MusicUGC-NER, a human-annotated dataset for detecting music entities (artists and works of art) in user-generated YouTube video titles, and combines it with the existing MusicRecoNER Reddit dataset. It benchmarks four LLMs (FireFunction-v2, GPT-4o-mini, Llama3.1-70B, Mixtral-8x22B) under in-context learning with tf-idf-based example retrieval against fine-tuned BERT and RoBERTa baselines, reporting strict F1 scores under five-fold cross-validation. In a second robustness study, the authors use GPT-4o-mini on a synthesized cloze dataset with entities from three groups (FMT-passed, FMT-failed, and 2024 post-cutoff debut artists from MusicBrainz) and two perturbation levels, analyzing error types. The paper claims that LLMs outperform SLMs in this ICL setting and that pre-training entity exposure has a large impact on detection performance, particularly for works-of-art.","tokens_in":17203,"tokens_out":9796,"duration_ms":79160,"significance":"The dataset and benchmark address an applied gap in music-domain NER over noisy user-generated content, and the authors should be credited for releasing the data, reporting high inter-annotator agreement (Kappa 0.93), and for the post-cutoff control with 2024 debut artists, which is a genuinely stronger unseen-entity manipulation than the FMT-based grouping. The perturbation taxonomy and the explicit matching-precision/recall evaluation are useful contributions. However, the FMT proxy for entity-string exposure is not validated, and the benchmark/robustness claims lack uncertainty quantification; these issues currently weaken the central causal conclusion. If the authors address these points, the paper would be a solid reference for practitioners choosing between fine-tuned SLMs and LLM-based ICL for music entity detection.","major_comments":[{"comment":"The FMT is used as the measure of pre-training entity exposure, but it tests canonical work–attribute knowledge (e.g., 'Who originally performed Yesterday?'), not the exact surface strings that appear in the D-YT video titles. A model can answer FMT questions through general world knowledge without having memorized abbreviated or misspelled YouTube-style utterances, and it can fail for reasons unrelated to the surface form. Therefore the .24 recall gap between Passed and Failed FMT groups in Table 4 may be confounded by entity frequency, length, or ambiguity, and the causal claim that exposure to the exact entity strings drives the drop is not established. The post-cutoff control in Section 4.2 is cleaner; the FMT-based interpretation should be reframed or supplemented with a string-level memorization probe (e.g., next-token prediction on the actual utterances).","section":"Section 4.1, Table 4"},{"comment":"Table 3 reports only mean F1 without standard deviations or significance tests. Differences such as GPT-4o-mini .84 vs Llama3.1-70B .82 are likely within fold-to-fold variation, so the claim that LLMs outperform SLMs is not yet quantified with confidence intervals. The FMT subsets in Table 4 have small, unbalanced supports (e.g., n=103 for GPT-4o-mini Failed vs n=229 Passed), making the .24 recall gap sensitive to a few instances. I request per-fold results, bootstrap intervals, or paired significance tests (e.g., McNemar) for the main comparisons.","section":"Section 6.1, Tables 3 and 4"},{"comment":"The robustness analysis plots error proportions without uncertainty bounds, and it is unclear whether the reported differences (e.g., the larger Post-Cutoff errors in Figure 3, or the perturbation-level trends in Figure 4) are reproducible or reflect sampling noise. Additionally, the six error categories are not independent: a misclassification contributes to both a missed and an incorrect count for the two classes, so the statement that 'the effect of exposure appears to be stronger than the effect of perturbation' is not supported by a statistical comparison. Please report error bars and a per-cloze paired comparison, or a single aggregate error measure.","section":"Section 6.2, Figures 3 and 4"},{"comment":"The ICL prompt is taken from a previous study on conversational Reddit queries and instructs the model to extract entities from 'a user request for music suggestions'; however, half of the benchmark data (D-YT) consists of YouTube video titles, which are not such requests. Using this prompt unmodified for YouTube titles may systematically disadvantage the LLMs relative to fine-tuned SLMs, which see task-native training data. The authors should either adapt the instruction to the YouTube setting or include a task-matched prompt control to rule out this confound.","section":"Section 5.2, Figure 8"}],"minor_comments":[{"comment":"Table 3, Mixtral-8x22B, k=0, D-RD+YT: the reported macro average .80 is inconsistent with the Artist .73 and WoA .67, whose mean is .70; this appears to be an arithmetic error.","section":"Table 3"},{"comment":"Table 4 caption: 'per outcome of the of the FMT' should be 'per outcome of the FMT'.","section":"Table 4 caption"},{"comment":"The perturbation generation uses p=0.5 but the exact distributions for character/word deletion, insertion, substitution and the interaction between Level-1 and Level-2 are only specified in the repository; for reproducibility, the paper should state these distributions, the random seed, and explicitly say whether abbreviations can be applied to WoA tokens.","section":"Section 4.2"},{"comment":"The term 'cloze' is used for masked templates that are filled with other entities, rather than for an infilling task; a brief clarification of this usage would avoid confusion.","section":"Section 4.2"}],"recommendation":"major_revision","confidential_remarks":"The paper is honest in its limitations and the dataset release is a plus. The main risk is the FMT-as-exposure interpretation; the post-cutoff control partially mitigates it, but the authors should avoid causal language about the FMT results without further validation. The numerical inconsistency in Table 3 should also be fixed."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper has one genuinely useful contribution and one that is half-proven. The genuinely useful part is the benchmark: a new annotated dataset of YouTube titles (D-YT), sensibly joined with MusicRecoNER, and a clean comparison of four LLMs under ICL against fine-tuned BERT/RoBERTa. The result that LLMs, especially GPT-4o-mini, outperform the SLMs on this UGC music NER task is credible and reproducible, given the released data and code. The annotation quality looks good (kappa 0.93), the preprocessing is documented carefully, and the tf-idf few-shot sampling is a sensible, if incremental, choice.\n\nThe robustness study is where I get cautious. The design is clever: build clozes from the annotated data, fill with entities stratified by a factual memorization test (FMT), add perturbations, and compare against Post-Cutoff entities. The finding that post-cutoff entities produce far more errors than perturbed entities is a strong, clean demonstration that exposure matters. But the FMT comparison itself is a soft spot. The FMT asks about original artist and composer using the canonical title, not the surface form that appears in a UGC utterance. Passing the FMT may simply reflect general world knowledge about a famous work, not memorization of the exact entity string used in the NER task. The paper treats FMT outcome as a proxy for exposure, and that conflation weakens the causal story. The group sizes are also small and imbalanced (e.g., 103 failures vs. 229 passes for GPT-4o-mini), and all results are reported without error bars or significance tests, so the 0.24 recall gap could easily be driven by confounds like entity frequency or ambiguity.\n\nThese are not fatal flaws. The benchmark stands on its own, and the post-cutoff control gives independent evidence that exposure has a real effect. But the abstract's claim that the paper \"uncovers the large impact of entity exposure\" is a bit ahead of what the FMT comparison alone supports. I would want the authors to either validate the FMT against a more direct exposure measure or reframe it as a secondary, suggestive analysis. I would also want to see variance across folds and some test for whether the FMT group differences are significant.\n\nWho is this for? Researchers in music information retrieval and anyone working on NER in noisy user-generated text will get value from the dataset and the benchmark. The robustness methodology is worth reading even if the exposure proxy needs work. It deserves a serious peer review, but the revision should address the proxy validity and statistical rigor before publication.","headline":"Solid new dataset and benchmark for music NER with LLMs; the exposure conclusion rests on a questionable proxy and needs tightening, but the core result holds.","tokens_in":17748,"tokens_out":2253,"would_cite":true,"duration_ms":23129,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"In-context-learning LLMs outperform fine-tuned small models on music entity detection, but the margin collapses for works the model did not memorize during pre-training.","keywords":["named entity recognition","music entity detection","in-context learning","large language models","entity exposure","factual memorization","user-generated content","robustness"],"falsifier":"Compare WoA recall across two groups of works that both pass the FMT but whose exact title strings have very different frequencies in typical training text (e.g., frequently covered singles versus obscure album tracks); if recall is identical, the effect is about the work rather than the string, and if recall tracks string frequency, the FMT is not the right exposure proxy.","tokens_in":16689,"feed_emoji":"🎵","tokens_out":9141,"duration_ms":72638,"temperature":0.7,"pith_summary":"This paper benchmarks four large language models against fine-tuned BERT and RoBERTa on a new music-entity dataset built from YouTube video titles and Reddit music-request posts, and then tests robustness to unseen and perturbed entity strings. The benchmark shows that LLMs used with in-context-learning reach higher macro F1 than the SLM baselines, with GPT-4o-mini the best at 0.84 with 35 few-shot examples. The second experiment reveals a catch: recall on works of art that a model fails on a two-question factual memorization test drops by up to 0.24, and controlled synthesis with never-before-seen post-cutoff entities produces the largest error rates. The paper concludes that part of the LLMs' apparent superiority is due to pre-training exposure to the exact musical works, not to a genuinely stronger ability to recognize novel entity strings.","feed_headline":"LLMs beat smaller models on music NER, yet exposure decides the win","feed_subtitle":"GPT-4o-mini's recall drops up to .24 for songs it fails a two-question memorization test.","key_machinery":"The paper's load-bearing machinery is a two-question Factual Memorization Test (FMT) built from the SHS100K cover-song knowledge base: for each musical work the model is asked who originally performed it and who wrote it, and the two answers are scored as Passed, Partial, or Failed. A second component is cloze-based data synthesis: 1,067 unique contexts are extracted from the annotated datasets, entities from three exposure groups (FMT Passed, FMT Failed, and post-cutoff debut artists from MusicBrainz) are inserted, and two levels of character/word perturbations and abbreviations are applied. The FMT is the proxy that ties NER recall back to pre-training exposure, while the cloze synthesis isolates context-driven detection from entity-driven recognition.","core_discovery":"The central claim is that LLMs with in-context-learning outperform fine-tuned SLMs on music entity detection in user-generated content, but that a substantial part of this advantage is an artifact of entity exposure: the better the model remembers the musical work from pre-training, the higher its recall on that work's title. The paper demonstrates this by splitting benchmark results according to a factual memorization test and by synthesizing three matched datasets whose entities are either known, partially known, or released after the model's knowledge cutoff. In those matched settings, error rates rise most sharply for the never-seen entities, and the effect of exposure is larger than the effect of realistic typos and abbreviations. Thus the authors argue that LLM-based music NER in the wild will inherit the same exposure bias long documented for smaller language models.","pith_inferences":["Replacing the FMT proxy with a direct string-level memorization probe (e.g., asking the model to complete the exact utterance) would sharpen the causal link the paper leaves implicit, separating 'knows the work' from 'has seen this exact string'.","The post-cutoff group gives a cleaner estimate of intrinsic NER ability than any in-distribution split, so reporting error rates on never-seen entities could become a standardized robustness score for future LLMs.","The same cloze-and-perturbation protocol transfers to other creative-content domains such as movies, books, or games, where entity names are non-regular and user text is noisy; the FMT would simply be replaced by a domain-appropriate factual probe.","Because the exposure effect outweighs context, adding an explicit 'unknown entity' option to the prompt might reduce spurious and incorrect labels on unseen works, a testable design change directly motivated by the paper's error analysis."],"forward_implications":["Practitioners using LLM-based music NER on long-tail catalogs should expect materially lower recall than benchmark numbers on popular hits suggest, because recall drops by up to 0.24 for works the model cannot recall.","Few-shot in-context-learning helps across all tested LLMs, so retrieval-based example selection (tf-idf) is preferable to zero-shot prompting for music entity detection.","Data augmentation with typos and abbreviations will not close the gap on genuinely unseen entities, since exposure has a stronger effect than perturbation; gazetteers or retrieval augmentation would be more promising.","Future NER benchmark comparisons should control for pre-training exposure, since the exposure effect documented for SLMs persists in LLMs and affects the ranking of models.","The released MusicUGC-NER dataset provides a shared, human-annotated IOB testbed that joins YouTube metadata and Reddit conversation, enabling direct comparison of music-domain NER in noisy user-generated text."],"supporting_citations":[{"why":"Supplies the SHS100K cover-song metadata with curated song-attribute links that the paper uses to build the factual memorization test and to match YouTube video titles.","marker":"Xu et al., 2018"},{"why":"Provides the MusicRecoNER Reddit dataset that is joined with the new YouTube data to form the benchmark, and the fine-tuning setup used for the SLM baselines.","marker":"Epure and Hennequin, 2023"},{"why":"Contributes the validated instruction prompt and the tf-idf few-shot example selection used for all LLM evaluations in the benchmark.","marker":"Hachmeier and Jäschke, 2024"},{"why":"Defines and characterizes memorization in general-purpose LLMs, motivating the construction of the factual memorization test.","marker":"Hartmann et al., 2023"},{"why":"Supplies the character-level and word-level perturbation taxonomy that the paper adapts for the robustness experiments.","marker":"Feng et al., 2024"},{"why":"Provides BERT, one of the two fine-tuned SLM baselines the LLMs are compared against.","marker":"Devlin et al., 2018"},{"why":"Provides RoBERTa, the other fine-tuned SLM baseline in the benchmark.","marker":"Liu et al., 2019"},{"why":"Articulates the contrary claim that LLMs are not good few-shot information extractors, which this paper's benchmark directly addresses with evidence of higher LLM performance.","marker":"Ma et al., 2023"}],"fun_headline_variants":["LLMs win music NER, but pre-training memory drives recall","In-context LLMs beat BERT on music entities, but exposure dominates","Music NER: LLMs top SLMs, but only for songs they remember","Exposure bias decides LLM win in music entity detection","LLM edge over SLMs in music NER depends on entity memory"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The central argument depends on the assumption that a two-question factual memorization test (who originally performed the song, who wrote it) is a valid measure of whether the model has memorized the exact entity string that appears in the NER task; if the test can be passed through reasoning or exposure to related entities, the measured recall drop cannot be attributed to pre-training exposure of that specific string.","fun_headline_variants_meta":{"raw":{"variants":["LLMs win music NER, but pre-training memory drives recall","In-context LLMs beat BERT on music entities, but exposure dominates","Music NER: LLMs top SLMs, but only for songs they remember","Exposure bias decides LLM win in music entity detection","LLM edge over SLMs in music NER depends on entity memory"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000847,"raw_usage":{"total_tokens":3652,"prompt_tokens":877,"completion_tokens":2775,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":493,"completion_tokens_details":{"reasoning_tokens":2680}},"tokens_in":493,"tokens_out":2775,"duration_ms":19760,"temperature":1.0,"reasoning_tokens":2680,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T14:30:44.709888+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compare WoA recall across two groups of works that both pass the FMT but whose exact title strings have very different frequencies in typical training text (e.g., frequently covered singles versus obscure album tracks); if recall is identical, the effect is about the work rather than the string, and if recall tracks string frequency, the FMT is not the right exposure proxy.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the MusicRecoNER Reddit dataset that is joined with the new YouTube data to form the benchmark, and the fine-tuning setup used for the SLM baselines."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Contributes the validated instruction prompt and the tf-idf few-shot example selection used for all LLM evaluations in the benchmark."}],"review_version":1}