REVIEW 6 cited by
Domain-Specific Language Model Pretraining for Biomedical Natural Language Processing
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Pretraining large neural language models, such as BERT, has led to impressive gains on many natural language processing (NLP) tasks. However, most pretraining efforts focus on general domain corpora, such as newswire and Web. A prevailing assumption is that even domain-specific pretraining can benefit by starting from general-domain language models. In this paper, we challenge this assumption by showing that for domains with abundant unlabeled text, such as biomedicine, pretraining language models from scratch results in substantial gains over continual pretraining of general-domain language models. To facilitate this investigation, we compile a comprehensive biomedical NLP benchmark from publicly-available datasets. Our experiments show that domain-specific pretraining serves as a solid foundation for a wide range of biomedical NLP tasks, leading to new state-of-the-art results across the board. Further, in conducting a thorough evaluation of modeling choices, both for pretraining and task-specific fine-tuning, we discover that some common practices are unnecessary with BERT models, such as using complex tagging schemes in named entity recognition (NER). To help accelerate research in biomedical NLP, we have released our state-of-the-art pretrained and task-specific models for the community, and created a leaderboard featuring our BLURB benchmark (short for Biomedical Language Understanding & Reasoning Benchmark) at https://aka.ms/BLURB.
Forward citations
Cited by 6 Pith papers
-
Auditing Retrieval-Augmented LLM Hypotheses for Longitudinal Cell Painting Morphology
A retrieval-augmented LLM framework generates evidence-linked biological hypotheses from longitudinal Cell Painting images, with automatic citation-integrity and morphology-compatibility audits.
-
Error-Aware Curriculum Learning for Biomedical Relation Classification
A teacher-student pipeline in which GPT-4o diagnoses a student's errors, assigns difficulty scores, and generates remediations, then trains a smaller model by curriculum learning, reports new state-of-the-art F1 on fo...
-
Chunk Twice, Embed Once: A Systematic Study of Segmentation and Representation Trade-offs in Chemistry-Aware Retrieval-Augmented Generation
A systematic evaluation shows that recursive 100-token non-overlapping chunks and retrieval-tuned embeddings outperform fixed-size chunks and domain-specific models like SciBERT for chemistry retrieval, and it introdu...
-
BIBERT-Pipe on Biomedical Nested Named Entity Linking at BioASQ 2025
A two-stage entity linking pipeline with boundary-cue tokens and data augmentation achieves third place on the BioNNE 2025 multilingual nested entity linking task.
-
The Next Phase of Scientific Fact-Checking: Advanced Evidence Retrieval from Complex Structured Academic Papers
The paper proposes a full-paper, evidence-aware retrieval agenda for scientific fact-checking and shows that combining verification feedback with semantic reranking improves evidence recall on SciFact-Open and Check-COVID.
-
Harnessing Collective Intelligence of LLMs for Robust Biomedical QA: A Multi-Model Approach
Combining 13 open-source LLMs by majority voting and answer union achieved top BioASQ 2025 rankings, but the reported best combinations are selected by exhaustive search and lack statistical validation.
Discussion (0). Sign in to comment.