{"id":"e2225092-56f5-46bd-8ea5-b58e224b7eb7","arxiv_id":"2605.28375","paper_version":1,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":6.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":0,"one_line_summary":"PrionNER is a new NER dataset of 317 prion-disease PubMed abstracts with 6,955 annotations across 15 coarse-grained and 31 fine-grained entity types, plus baseline benchmarks.","lead":"This paper releases PrionNER, a new manually annotated dataset of 317 PubMed abstracts focused on prion diseases, containing 6,955 entity mentions across 15 coarse and 31 fine clinical categories. It supplies a benchmark for rare-disease information extraction where existing resources are scarce.","discovery_kind":"new_application","skeptic_critique":{"model":"grok-4.3","headline":"No significant objection identified","rationale":"The reader's weakest assumption correctly flags a practical limitation for users of the dataset, but the paper itself makes no stronger claim than having produced and released the resource with the stated statistics. Because the core assertion is descriptive rather than inferential, no load-bearing internal inconsistency or unsupported technical step is visible from the given material.","tokens_in":1812,"tokens_out":226,"duration_ms":25893,"concrete_test":"Download the released dataset and annotation guidelines from the GitHub link; verify that the 6,955 annotations and 31 fine-grained types match the counts and definitions stated in the abstract.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is simply that PrionNER exists as a manually annotated NER dataset with the reported size, entity inventory, IAA, and baseline results. The provided abstract and dataset release satisfy this descriptive claim on its face. The representativeness concern noted by the reader is real for downstream utility but is not required for the paper to have delivered what it claims to deliver.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.3","summary":"The paper introduces PrionNER, a manually annotated named entity recognition dataset for prion disease clinical information in PubMed abstracts. The current release comprises 317 abstracts, 2,943 sentences, and 6,955 text-bound entity annotations spanning 15 coarse-grained and 31 fine-grained clinically oriented entity types covering diseases, symptoms, diagnostics, findings, anatomy, treatments, and temporal and statistical evidence. Inter-annotator agreement reaches 81.78 exact-match F1. The paper benchmarks supervised BERT baselines, W2NER, and zero-shot extractors, finding W2NER strongest among supervised models and Gemma-4-31B strongest among zero-shot models, while noting the benchmark remains challenging for complex mentions and fine-grained distinctions. The dataset, guidelines, and scripts are publicly released.","tokens_in":1880,"tokens_out":498,"duration_ms":41291,"significance":"If the reported annotation quality and entity coverage hold, PrionNER fills a documented gap as the first publicly available prion-disease-focused NER resource. The combination of manual annotation, high IAA, clinical entity granularity, and public release of data plus evaluation scripts constitutes a concrete, reusable contribution to rare-disease biomedical NLP under low-resource and fine-grained conditions. The baseline results further establish the dataset as a non-trivial benchmark.","major_comments":[],"minor_comments":[{"comment":"The abstract asserts that 'to our knowledge, there is no publicly available prion-disease-focused dataset'; the introduction should include a short, explicit comparison to existing biomedical NER corpora (e.g., BC5CDR, NCBI Disease) to substantiate this claim rather than leaving it as an assertion.","section":"Introduction"},{"comment":"Section describing the entity inventory should clarify the relationship between the 15 coarse-grained and 31 fine-grained types (e.g., whether the fine-grained labels are strict subtypes or include additional distinctions) and provide at least one example sentence per major category to aid reader comprehension.","section":"Dataset Construction"},{"comment":"The results section notes that the benchmark is 'challenging, especially for structurally complex mentions and fine-grained clinically adjacent label distinctions' but provides no quantitative breakdown or illustrative examples; adding a short error-analysis paragraph or table would strengthen the presentation of the baseline findings.","section":"Experiments and Results"}],"recommendation":"minor_revision","confidential_remarks":null},"author_rebuttal":{"model":"grok-4.3","summary":"We thank the referee for the positive summary of PrionNER and the recommendation of minor revision. The assessment correctly identifies the dataset's scope, annotation quality, and benchmark results. No major comments were provided in the report.","responses":[],"tokens_in":1347,"tokens_out":63,"duration_ms":10740,"standing_objections":[]},"desk_editor":{"model":"grok-4.3","letter":"This paper's main contribution is releasing PrionNER, the first publicly available NER dataset focused on prion disease literature from PubMed abstracts. It includes 317 abstracts, 2943 sentences, 6955 entity annotations with 15 coarse and 31 fine-grained types, and reports 81.78 exact-match F1 for inter-annotator agreement. They also provide baseline results with models like W2NER and zero-shot LLMs.\n\nThe work does a good job documenting the annotation process at a high level and making the dataset, guidelines, and evaluation scripts available on GitHub. The entity types are clinically oriented, which fits the goal of supporting rare-disease biomedical NLP.\n\nThe soft spots are mostly around scope and depth. The dataset is restricted to abstracts, which may not capture the full complexity of the literature. At 317 documents it's small for training modern models, and the paper does not appear to include detailed sampling methodology or extensive error analysis beyond noting that complex mentions and fine-grained distinctions are challenging. The baselines are standard, so they confirm the task is hard but do not push the state of the art.\n\nThis is for specialists in biomedical information extraction who need data for underrepresented diseases or fine-grained clinical entities. It would be useful in a reading group focused on dataset papers or rare disease NLP.\n\nThe paper shows clear thinking in creating and validating the resource. I would send it for peer review because the dataset is new and the reported figures are concrete.","headline":"PrionNER is the first public NER dataset for prion disease abstracts, with decent IAA and released resources, but small scale and abstract-only limits its reach.","tokens_in":2330,"tokens_out":377,"would_cite":false,"duration_ms":31170,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.3","headline":"PrionNER supplies the first manually annotated NER dataset for prion disease entities in 317 PubMed abstracts.","keywords":["PrionNER","named entity recognition","prion disease","biomedical literature","PubMed abstracts","information extraction","rare disease NLP","clinical entity annotation"],"falsifier":"A comparison showing that models trained on PrionNER extract no more useful clinical entities from held-out prion literature or real patient records than models trained only on general biomedical NER corpora.","tokens_in":2727,"feed_emoji":"📄","tokens_out":695,"duration_ms":15844,"temperature":0.7,"pith_summary":"The paper creates PrionNER to fill the gap in publicly available resources for extracting clinical information from prion disease literature. It manually labels 6955 text spans across 15 coarse and 31 fine entity types that include diseases, symptoms, diagnostics, findings, anatomy, treatments, and temporal evidence. Inter-annotator agreement reaches 81.78 exact-match F1, and benchmarks with BERT-style models, W2NER, and zero-shot extractors show the task remains difficult for complex mentions and fine-grained distinctions. The dataset is positioned as a benchmark for rare-disease biomedical NLP under low-resource and non-flat extraction settings.","feed_headline":"PrionNER dataset labels 6955 entities across 317 prion abstracts","feed_subtitle":"New resource supplies 15 coarse and 31 fine clinical types to benchmark extraction from rare-disease literature.","key_machinery":"The PrionNER dataset with its 15/31 entity type inventory and annotation guidelines for clinical information in prion disease abstracts.","core_discovery":"PrionNER is a new dataset of 317 PubMed abstracts containing 2943 sentences and 6955 entity annotations that span 15 coarse-grained and 31 fine-grained clinically oriented types covering diseases, symptoms, diagnostics, findings, anatomy, treatments, and temporal and statistical evidence, with 81.78 exact-match F1 inter-annotator agreement, and it serves as a benchmark where W2NER is the strongest supervised model and Gemma-4-31B the strongest zero-shot model.","pith_inferences":["The dataset could be used to measure how well general biomedical language models transfer to narrow disease domains without additional fine-tuning.","Extending the annotations from abstracts to full-text articles would test whether the same entity inventory remains sufficient outside the abstract format.","Linking the extracted entities to existing medical ontologies might reveal gaps in how prion-specific findings are represented in broader knowledge bases."],"forward_implications":["Supervised models can be trained directly on the 6955 annotations to extract the listed clinical entity types from new prion abstracts.","Zero-shot models can be evaluated against the fine-grained distinctions that separate adjacent clinical categories.","The resource supports development of information extraction systems tailored to rare neurodegenerative diseases.","The annotation scheme provides a template for similar datasets on other low-prevalence conditions.","Benchmark results indicate that current architectures still need improvement on structurally complex or context-dependent mentions."],"fun_headline_variants":["PrionNER annotates 6955 entities in 317 prion abstracts","PrionNER covers 15 coarse and 31 fine clinical entity types","PrionNER reaches 81.78 F1 inter-annotator agreement","W2NER leads supervised models on PrionNER benchmark"],"cache_read_input_tokens":2112,"weakest_assumption_plain":"The selected PubMed abstracts and the chosen entity type categories capture the clinically relevant information that practitioners actually need from prion disease literature.","fun_headline_variants_meta":{"raw":{"variants":["PrionNER annotates 6955 entities in 317 prion abstracts","PrionNER covers 15 coarse and 31 fine clinical entity types","PrionNER reaches 81.78 F1 inter-annotator agreement","W2NER leads supervised models on PrionNER benchmark"]},"model":"grok-4.3","cost_usd":0.008307,"raw_usage":{"total_tokens":3798,"prompt_tokens":735,"num_sources_used":0,"completion_tokens":75,"cost_in_usd_ticks":83074500,"prompt_tokens_details":{"text_tokens":735,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":2988,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":735,"tokens_out":75,"duration_ms":27700,"temperature":1.0,"reasoning_tokens":2988,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-06-29T12:45:01.428441+00:00","model_set":{"reader":"grok-4.3"},"falsifier":"A comparison showing that models trained on PrionNER extract no more useful clinical entities from held-out prion literature or real patient records than models trained only on general biomedical NER corpora.","supporting_citations":[],"review_version":1}