Pith. sign in

REVIEW 4 major objections 5 minor 32 references

BERT-based Ranking for Biomedical Entity Normalization

T0 review · 4 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash

Pith's one-line read Fine-tuning pre-trained BERT-family models for biomedical entity normalization consistently beats previous state-of-the-art systems on three datasets, with up to 1.17% accuracy gain.

desk verdict A useful, competent first application of BERT to biomedical entity normalization, with modest gains that hold on two datasets but a shaky NCBI comparison and unresolved model-selection details. read the letter →

arxiv 1908.03548 v1 pith:R4JRT6I4 submitted 2019-08-09 cs.IR cs.CLcs.LG

classification cs.IRcs.CLcs.LG
keywords biomedicalentitynormalizationlinkingBERTBioClinicalcandidateconceptrankingsentence-pairclassificationtermvariation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper seeks to show that pre-trained bidirectional language models can be adapted to biomedical entity normalization, the task of linking an entity mention in text to a standard concept identifier. The authors fine-tune BERT, BioBERT, and ClinicalBERT as a pointwise ranker over candidate concepts and report accuracy of 91.10% on ShARe/CLEF clinical notes, 89.06% on NCBI PubMed abstracts, and 93.22% on TAC2017ADR drug labels. These numbers beat the previously published best systems by 0.35, 0.26, and 1.17 percentage points, respectively. The result matters because term variation, the presence of many surface forms for one concept, is the dominant difficulty in biomedical normalization, and contextualized representations directly address that difficulty.

What carries the argument

The load-bearing component is a pointwise ranking module built by fine-tuning a pre-trained BERT-family model, where BERT is a bidirectional Transformer language model trained on large text corpora. For each mention $m$ and candidate concept $c$, the input is the token sequence $[CLS]\,m\,[SEP]\,c$; the final hidden state of $[CLS]$ goes through a newly added softmax layer $W$, and the predicted probability that the pair is a correct link serves as the ranking score. Candidate concepts come from Lucene BM25, and a NIL threshold learned on held-out training data decides when the top-ranked concept should be rejected as unlinkable. This transfers contextualized word representations from general, biomedical, or clinical corpora to the normalization task without changing the underlying model.

What would settle it

Run the same fine-tuning grid on the three datasets but choose batch size and epoch count using a development split only, and report test accuracy at every grid point. If the dev-selected model does not achieve the best test accuracy, or its test accuracy drops to the previous baseline levels, the claimed improvement is an artifact of selection.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that a simple sentence-pair classification architecture, fed by BM25 candidate retrieval, transfers pre-trained language representations to biomedical entity normalization more effectively than earlier feature-based, rule-based, or CNN systems. The best fine-tuned model is BioBERTBase_Cased+PubMed on ShARe/CLEF and TAC2017ADR and BioBERTBase_Cased+PubMed+PMC on NCBI; both consistently outperform the BM25 baseline and the reimplemented CNN ranking baseline. The paper also reports that cased BERT models beat uncased versions in most comparisons, and that domain-specific BioBERT and ClinicalBERT generally outperform generic BERT on clinical and drug-label text. On NCBI the improvement over the joint TaggerOne system is only 0.26%, which the authors attribute to the difficulty of that dataset and the joint-model advantage of the baseline.

Load-bearing premise

The reported gains stand on the premise that the best epoch and batch size were chosen on a held-out development set; if the test set guided those choices, the improvements are inflated.

Editorial extensions

If this is right

  • The paper's results imply that pre-trained language models should replace context-independent word embeddings as the default representation for biomedical entity normalization.
  • Domain-matched pre-training helps: BioBERT and ClinicalBERT outperform generic BERT on clinical notes and drug labels, so choosing a pre-training corpus to match the target text is a practical lever.
  • Cased variants are usually better than uncased variants, indicating that case information is useful for biomedical concept normalization.
  • Because the largest gain appears on TAC2017ADR drug labels, the method is likely to be most valuable where term variation is severe and existing systems are weaker.
  • The authors state that pairwise learning-to-rank and joint recognition-normalization models are natural extensions, implying the current pointwise setting is not the ceiling of the approach.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the claimed gains hold under dev-based model selection, the same BM25-plus-fine-tuning recipe should transfer to other biomedical entity types such as chemicals, genes, and procedures, where term variation is common.
  • The small margins on ShARe/CLEF and NCBI suggest that candidate recall from BM25 may be the binding constraint, so enlarging or improving candidate generation could yield larger improvements than further model tuning.
  • The softmax score produced for each pair could be reused as a calibrated confidence for unlinkable-mention prediction, potentially removing the need for a separately learned NIL threshold.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The manuscript proposes an entity normalization architecture in which BM25 candidate generation is followed by fine-tuning of BERT, BioBERT, and ClinicalBERT models as a binary mention–concept classifier, with a learned threshold for unlinkable mentions. The authors evaluate on ShARe/CLEF clinical notes, NCBI disease abstracts, and TAC2017ADR drug labels, comparing nine pre-trained models against BM25, a reimplemented CNN baseline, and published results from UWM, TaggerOne, Xu et al., and D'Souza and Ng. They report consistent state-of-the-art accuracy, with gains of 0.35%, 0.26%, and 1.17% on the three datasets. The central contribution is empirical: a demonstration that fine-tuned contextualized representations improve biomedical entity normalization.

Significance. If the reported gains are real, the paper provides a useful baseline for biomedical entity normalization and is among the first to apply BERT-family models to this task. The architecture is simple and clearly described, the coverage of nine pre-trained models is extensive, and the inclusion of a reimplemented CNN baseline on the same reconstructed knowledge bases is a strength. However, the incremental gains are very small on two of the three datasets, and the central claim rests on the trustworthiness of these small differences. The contribution is empirical rather than methodological, and no code or trained models are released.

major comments (4)
  1. [Parameters Settings; Table 4] The hyperparameter description states that batch size (16, 32) and training epochs (1 through 10) were tuned and "the model with the best performance" was saved, but it does not specify which data split was used for this selection. Since Tables 2 and 4 report test-set accuracies, the results as presented are consistent with model selection on the test set. With 20 hyperparameter configurations per model, test-set selection would inflate the reported accuracy and make the 0.26–1.17% gains uninterpretable. Please select epochs and batch size on a held-out development set and then report the test accuracy of the chosen configuration, or provide a correction for selection bias.
  2. [NCBI dataset; Table 3] The best NCBI model, BioBERTBase_Cased+PubMed+PMC, was pre-trained on PubMed abstracts and PMC full text, while the NCBI test set consists of 100 PubMed abstracts that are very likely contained in that pre-training corpus. Because the 0.26% gain over TaggerOne corresponds to roughly 2.5 mentions out of 960, even modest memorization of token-level patterns from the test abstracts could change the comparison. The paper neither reports whether the NCBI test abstracts overlap with BioBERT's pre-training data nor provides a control. Please quantify the overlap and rerun the NCBI comparison with a model that has not seen the test abstracts, or otherwise demonstrate that the result is unaffected.
  3. [Table 3; Comparisons with existing work] The reported differences on ShARe/CLEF and NCBI correspond to roughly 19 and 2.5 test mentions, respectively. No confidence intervals, bootstrap estimates, or significance tests are provided, and fine-tuning is stochastic, so the observed gaps could lie within run-to-run or sampling noise. Please report repeated runs or bootstrap confidence intervals (at least exact mention counts) and state whether the differences are statistically reliable.
  4. [Table 3; Comparisons with existing work] The comparison mixes problem settings and knowledge-base versions. TaggerOne is a joint NER+normalization model while the proposed system is normalization-only with gold mentions, and the UWM and D'Souza and Ng scores were obtained with their original knowledge bases while the proposed model uses a reconstructed SNOMED-CT subset. The paper acknowledges these differences but still claims "consistently outperformed." To support the claim, the comparisons should either use exactly the same evaluation protocol and knowledge bases, or report reimplementations (as was done for the CNN) for all baselines. At minimum, clarify whether the TaggerOne 88.80 result is normalization-only with gold mentions or an end-to-end NER+normalization result.
minor comments (5)
  1. [Parameters Settings] The phrase "saved the model with the best performance" should identify the evaluation metric and the split on which the choice was made; as written it is ambiguous and also hampers reproducibility.
  2. [Parameters Settings] The paper does not report the maximum sequence length, optimizer details, learning-rate schedule, or number of random seeds; these details are needed to reproduce the fine-tuning results.
  3. [Discussion] The statement that "BERT models with cased version were better than the uncased version in most cases" should be qualified with respect to TAC2017ADR, where the uncased variants outperform or tie the cased variants in Table 2.
  4. [Preprocessing; System Architecture] The preprocessing pipeline lowercases, stemms, and removes punctuation before fine-tuning, but BERT's WordPiece tokenizer has its own tokenization; please state explicitly whether the preprocessed mention is fed directly to BERT or re-tokenized, since this affects the validity of the cased-versus-uncased comparison.
  5. [Abstract and Notation] There are minor presentation issues: "we proposed" in the abstract should be "we propose," and the mathematical notation for the ranking score and candidate set is corrupted in the rendered text (e.g., 𝑠𝑐𝑜𝑟𝑒(𝑚,𝑐)=𝑠𝑜𝑓𝑡𝑚𝑎𝑥(𝐶𝑊P) and {𝑊Y}YZ[[\).

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the reported results are measured accuracies on external benchmark test sets, with no load-bearing step that reduces to the paper's own inputs.

full rationale

This paper is an empirical evaluation, not a derivation: the central claim is that fine-tuning pre-trained BERT-family models improves entity-normalization accuracy on three benchmark datasets. Accuracy is computed on held-out test sets against external baselines, and the gains are measured, not fitted. The candidate ranking model is a standard sentence-pair classification fine-tuning procedure, and the NIL threshold is learned from a held-out development subset, which is a normal experimental design rather than a circular construction. The paper's use of self-citations is not load-bearing: the ClinicalBERT models cited from the same group are publicly released pre-trained models, the TAC2017ADR baseline from Xu et al. is a published challenge system, and none of these citations are invoked to forbid alternatives or to define the target result. The acknowledged limitations about not fully reconstructing prior KBs and the ambiguity of 'saved the model with the best performance' are methodological and reproducibility concerns about selection and comparability, not circularity, since the reported numbers still come from test-set evaluation and are not algebraically entailed by the hyperparameters or by any equation in the paper. No quoted step in the manuscript exhibits a reduction of a prediction to a fitted input or a self-citation chain that forces the conclusion, so no circularity is present.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The paper does not introduce new theoretical entities. It relies on several domain assumptions about dataset quality, BM25 recall, and transferability of pre-trained models. The main free parameters are hyperparameters for fine-tuning and the NIL threshold; the lack of explicit separation between development and test selection is the most consequential.

free parameters (5)
  • number of training epochs = 1 to 10, selected by best performance
    The paper tunes epochs from 1 to 10 and saves the model with the best performance, but does not state whether the selection is on a development or test split. This directly affects the reported accuracy.
  • batch size = 16 or 32, selected per dataset
    Batch size is tuned between 16 and 32; Table 4 shows the chosen value varies by dataset and model. The selection criterion is not specified.
  • NIL threshold tau = Learned from training data with a held-out development set
    The threshold for deciding whether the top-ranked concept is a valid link rather than NIL is learned from data. It is a free parameter that affects accuracy.
  • learning rate = 2e-5
    The learning rate is fixed at 2e-5, a standard value for BERT fine-tuning, but it is still a hyperparameter choice that affects results.
  • BM25 top-k candidates = 10
    The candidate concept generation retrieves the top 10 candidates per mention. The choice of k is not varied and affects the upper bound on recall.
assumptions (5)
  • domain assumption Gold-standard annotations in the three datasets are correct and consistent.
    The evaluation treats the ShARe/CLEF, NCBI, and TAC2017ADR annotations as ground truth for all accuracy measurements.
  • domain assumption The pre-trained BERT, BioBERT, and ClinicalBERT models provide useful representations for biomedical text.
    The paper assumes that fine-tuning these external models is appropriate for entity normalization, which is supported by prior work but still an assumption about transferability.
  • domain assumption BM25 candidate generation recalls the correct concept within the top 10 candidates for all linkable mentions.
    The ranker can only reorder candidates returned by BM25. If the correct concept is not in the top 10, the system fails regardless of the BERT ranker.
  • ad hoc to paper The reconstructed knowledge bases (UMLS subset, MEDIC, MedDRA) are close enough to those used in prior work for fair comparison.
    The authors acknowledge that they may not have completely reconstructed the ontology used in prior work (e.g., Li et al.), which could affect the comparison.
  • domain assumption Accuracy, the percentage of correctly normalized mentions, is an appropriate evaluation metric for this task.
    Accuracy is used throughout; it treats all mentions equally and does not distinguish between unlinkable and linkable mentions.

how reviews work

0 comments
Cite this review

Pith. "Pith review of BERT-based Ranking for Biomedical Entity Normalization." pith.science (2026). https://pith.science/paper/R4JRT6I4

@misc{pith2026190803548,
  author       = {Pith},
  title        = {Pith review of: BERT-based Ranking for Biomedical Entity Normalization},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/R4JRT6I4}},
  note         = {Machine review of arXiv:1908.03548}
}
read the original abstract

Developing high-performance entity normalization algorithms that can alleviate the term variation problem is of great interest to the biomedical community. Although deep learning-based methods have been successfully applied to biomedical entity normalization, they often depend on traditional context-independent word embeddings. Bidirectional Encoder Representations from Transformers (BERT), BERT for Biomedical Text Mining (BioBERT) and BERT for Clinical Text Mining (ClinicalBERT) were recently introduced to pre-train contextualized word representation models using bidirectional Transformers, advancing the state-of-the-art for many natural language processing tasks. In this study, we proposed an entity normalization architecture by fine-tuning the pre-trained BERT / BioBERT / ClinicalBERT models and conducted extensive experiments to evaluate the effectiveness of the pre-trained models for biomedical entity normalization using three different types of datasets. Our experimental results show that the best fine-tuned models consistently outperformed previous methods and advanced the state-of-the-art for biomedical entity normalization, with up to 1.17% increase in accuracy.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

32 extracted references · 25 canonical work pages

  1. [1]

    unmapped

    BERT-based Ranking for Biomedical Entity Normalization Zongcheng Ji, PhD1, Qiang Wei, MS1, Hua Xu, PhD1 1School of Biomedical Informatics, The University of Texas Health Science Center at Houston, Houston, TX, USA Abstract Developing high-performance entity normalization algorithms that can alleviate the term variation problem is of great interest to the ...

  2. [2]

    Entity Linking with a Knowledge Base: Issues, Techniques, and Solutions

    Shen W, Wang J, Han J. Entity Linking with a Knowledge Base: Issues, Techniques, and Solutions. TKDE. 2014;99:1. doi:http://doi.ieeecomputersociety.org/10.1109/TKDE.2014.2327028

  3. [3]

    CNN-based ranking for biomedical entity normalization

    Li H, Chen Q, Tang B, et al. CNN-based ranking for biomedical entity normalization. BMC Bioinformatics. 2017;18(11):385

  4. [4]

    Sieve-Based Entity Linking for the Biomedical Domain

    D’Souza J, Ng V. Sieve-Based Entity Linking for the Biomedical Domain. In: ACL. ; 2015:297-302

  5. [5]

    UWM: Disorder Mention Extraction from Clinical Text Using CRFs and Normalization Using Learned Edit Distance Patterns

    Ghiasvand O, Kate RJ. UWM: Disorder Mention Extraction from Clinical Text Using CRFs and Normalization Using Learned Edit Distance Patterns. In: SemEval@COLING. ; 2014:828-832

  6. [6]

    Using rule-based natural language processing to improve disease normalization in biomedical text

    Kang N, Singh B, Afzal Z, van Mulligen EM, Kors JA. Using rule-based natural language processing to improve disease normalization in biomedical text. JAMIA. 2012;20(5):876-881

  7. [7]

    DNorm: disease name normalization with pairwise learning to rank

    Leaman R, Doǧan RI, Lu Z. DNorm: disease name normalization with pairwise learning to rank. Bioinformatics. 2013;29:2909-2917

  8. [9]

    The Unified Medical Language System (UMLS): integrating biomedical terminology

    Bodenreider O. The Unified Medical Language System (UMLS): integrating biomedical terminology. Nucleic Acids Res. 2004;32(suppl_1):D267. doi:10.1093/nar/gkh061

Show all 32 references
  1. [10]

    SemEval-2014 Task 7: Analysis of Clinical Text

    Pradhan S, Elhadad N, Chapman WW, Manandhar S, Savova G. SemEval-2014 Task 7: Analysis of Clinical Text. In: SemEval. ; 2014:54-62

  2. [11]

    Large-scale linear ranksvm

    Lee C-P, Lin C-J. Large-scale linear ranksvm. Neural Comput. 2014;26(4):781-817

  3. [12]

    Overview of the TAC 2017 Adverse Reaction Extraction from Drug Labels Track

    Roberts K, Demner-Fushman D, Tonning JM. Overview of the TAC 2017 Adverse Reaction Extraction from Drug Labels Track. In: TAC

  4. [14]

    Distributed representations of words and phrases and their compositionality

    Mikolov T, Sutskever I, Chen K, Corrado GS, Dean J. Distributed representations of words and phrases and their compositionality. In: Advances in Neural Information Processing Systems. ; 2013:3111-3119

  5. [15]

    Deep Contextualized Word Representations

    Peters ME, Neumann M, Iyyer M, et al. Deep Contextualized Word Representations. In: NAACL-HLT. ; 2018:2227-2237. https://aclanthology.info/papers/N18-1202/n18-1202

  6. [18]

    BioBERT: a pre-trained biomedical language representation model for biomedical text mining

    Lee J, Yoon W, Kim S, et al. BioBERT: a pre-trained biomedical language representation model for biomedical text mining. CoRR. 2019;abs/1901.0. http://arxiv.org/abs/1901.08746

  7. [19]

    Enhancing clinical concept extraction with contextual embeddings

    Si Y, Wang J, Xu H, Roberts K. Enhancing clinical concept extraction with contextual embeddings. JAMIA. July 2019:ocz096. doi:10.1093/jamia/ocz096

  8. [20]

    ClinicalBERT: Modeling Clinical Notes and Predicting Hospital Readmission

    Huang K, Altosaar J, Ranganath R. ClinicalBERT: Modeling Clinical Notes and Predicting Hospital Readmission. CoRR. 2019;abs/1904.0. http://arxiv.org/abs/1904.05342

  9. [21]

    Publicly Available Clinical BERT Embeddings

    Alsentzer E, Murphy JR, Boag W, et al. Publicly Available Clinical BERT Embeddings. CoRR. 2019;abs/1904.0. http://arxiv.org/abs/1904.03323

  10. [23]

    Evaluating the state of the art in disorder recognition and normalization of the clinical narrative

    Pradhan S, Elhadad N, South BR, et al. Evaluating the state of the art in disorder recognition and normalization of the clinical narrative. JAMIA. 2015;22(1):143-154. doi:10.1136/amiajnl-2013-002544

  11. [24]

    NCBI disease corpus: a resource for disease name recognition and concept normalization

    Doǧan RI, Leaman R, Lu Z. NCBI disease corpus: a resource for disease name recognition and concept normalization. JBI. 2014;47:1-10

  12. [25]

    MEDIC: a practical disease vocabulary used at the Comparative Toxicogenomics Database

    Davis AP, Wiegers TC, Rosenstein MC, Mattingly CJ. MEDIC: a practical disease vocabulary used at the Comparative Toxicogenomics Database. Database. 2012;2012:bar065. doi:10.1093/database/bar065

  13. [26]

    Abbreviation definition identification based on automatic precision estimates

    Sohn S, Comeau DC, Kim W, Wilbur WJ. Abbreviation definition identification based on automatic precision estimates. BMC Bioinformatics. 2008;9. doi:10.1186/1471-2105-9-402

  14. [27]

    A Simple Algorithm for Identifying Abbreviation Definitions in Biomedical Text

    Schwartz AS, Hearst MA. A Simple Algorithm for Identifying Abbreviation Definitions in Biomedical Text. In: Proceedings of the 8th Pacific Symposium on Biocomputing. ; 2003:451-462

  15. [28]

    CLAMP - a toolkit for efficiently building customized clinical natural language processing pipelines

    Soysal E, Wang J, Jiang M, et al. CLAMP - a toolkit for efficiently building customized clinical natural language processing pipelines. JAMIA. 2017;25(3):331-336

  16. [29]

    An Information Retrieval Approach to Short Text Conversation

    Ji Z, Lu Z, Li H. An Information Retrieval Approach to Short Text Conversation. http://arxiv.org/abs/14086988

  17. [30]

    UTH-CCB: The Participation of the SemEval 2015 Challenge - Task

    Xu J, Zhang Y, Wang J, et al. UTH-CCB: The Participation of the SemEval 2015 Challenge - Task

  18. [31]

    2015:311-314

    In: SemEval. ; 2015:311-314. http://www.aclweb.org/anthology/S15-2052

  19. [32]

    UTH_CCB: a report for semeval 2014 - task 7 analysis of clinical text

    Zhang Y, Wang J, Tang B, et al. UTH_CCB: a report for semeval 2014 - task 7 analysis of clinical text. In: SemEval. ; 2014:802

  20. [33]

    Okapi at TREC-3

    Robertson SE, Walker S, Jones S, Hancock-Beaulieu M, Gatford M. Okapi at TREC-3. In: Proceedings of TREC. ; 1995:109-126

  21. [34]

    BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

    Devlin J, Chang M-W, Lee K, Toutanova K. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. CoRR. 2018;abs/1810.0. http://arxiv.org/abs/1810.04805

  22. [35]

    MIMIC-III, a freely accessible critical care database

    Johnson AEW, Pollard TJ, Shen L, et al. MIMIC-III, a freely accessible critical care database. Sci data. 2016;3:160035

  23. [36]

    TaggerOne: joint named entity recognition and normalization with semi-Markov Models

    Leaman R, Lu Z. TaggerOne: joint named entity recognition and normalization with semi-Markov Models. Bioinformatics. 2016;32(18):2839-2846

  24. [38]

    Joint Recognition and Linking of Fine-Grained Locations from Tweets

    Ji Z, Sun A, Cong G, Han J. Joint Recognition and Linking of Fine-Grained Locations from Tweets. In: WWW. ; 2016:1271-1281. doi:10.1145/2872427.2883067

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.