REVIEW 4 major objections 5 minor 32 references
BERT-based Ranking for Biomedical Entity Normalization
T0 review · 4 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash
Pith's one-line read Fine-tuning pre-trained BERT-family models for biomedical entity normalization consistently beats previous state-of-the-art systems on three datasets, with up to 1.17% accuracy gain.
desk verdict A useful, competent first application of BERT to biomedical entity normalization, with modest gains that hold on two datasets but a shaky NCBI comparison and unresolved model-selection details. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing component is a pointwise ranking module built by fine-tuning a pre-trained BERT-family model, where BERT is a bidirectional Transformer language model trained on large text corpora. For each mention $m$ and candidate concept $c$, the input is the token sequence $[CLS]\,m\,[SEP]\,c$; the final hidden state of $[CLS]$ goes through a newly added softmax layer $W$, and the predicted probability that the pair is a correct link serves as the ranking score. Candidate concepts come from Lucene BM25, and a NIL threshold learned on held-out training data decides when the top-ranked concept should be rejected as unlinkable. This transfers contextualized word representations from general, biomedical, or clinical corpora to the normalization task without changing the underlying model.
What would settle it
Run the same fine-tuning grid on the three datasets but choose batch size and epoch count using a development split only, and report test accuracy at every grid point. If the dev-selected model does not achieve the best test accuracy, or its test accuracy drops to the previous baseline levels, the claimed improvement is an artifact of selection.
Extended reading notes
Core claim
On the paper's own terms, the discovery is that a simple sentence-pair classification architecture, fed by BM25 candidate retrieval, transfers pre-trained language representations to biomedical entity normalization more effectively than earlier feature-based, rule-based, or CNN systems. The best fine-tuned model is BioBERTBase_Cased+PubMed on ShARe/CLEF and TAC2017ADR and BioBERTBase_Cased+PubMed+PMC on NCBI; both consistently outperform the BM25 baseline and the reimplemented CNN ranking baseline. The paper also reports that cased BERT models beat uncased versions in most comparisons, and that domain-specific BioBERT and ClinicalBERT generally outperform generic BERT on clinical and drug-label text. On NCBI the improvement over the joint TaggerOne system is only 0.26%, which the authors attribute to the difficulty of that dataset and the joint-model advantage of the baseline.
Load-bearing premise
The reported gains stand on the premise that the best epoch and batch size were chosen on a held-out development set; if the test set guided those choices, the improvements are inflated.
Editorial extensions
If this is right
- The paper's results imply that pre-trained language models should replace context-independent word embeddings as the default representation for biomedical entity normalization.
- Domain-matched pre-training helps: BioBERT and ClinicalBERT outperform generic BERT on clinical notes and drug labels, so choosing a pre-training corpus to match the target text is a practical lever.
- Cased variants are usually better than uncased variants, indicating that case information is useful for biomedical concept normalization.
- Because the largest gain appears on TAC2017ADR drug labels, the method is likely to be most valuable where term variation is severe and existing systems are weaker.
- The authors state that pairwise learning-to-rank and joint recognition-normalization models are natural extensions, implying the current pointwise setting is not the ceiling of the approach.
Reading between the lines
- If the claimed gains hold under dev-based model selection, the same BM25-plus-fine-tuning recipe should transfer to other biomedical entity types such as chemicals, genes, and procedures, where term variation is common.
- The small margins on ShARe/CLEF and NCBI suggest that candidate recall from BM25 may be the binding constraint, so enlarging or improving candidate generation could yield larger improvements than further model tuning.
- The softmax score produced for each pair could be reused as a calibrated confidence for unlinkable-mention prediction, potentially removing the need for a separately learned NIL threshold.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The manuscript proposes an entity normalization architecture in which BM25 candidate generation is followed by fine-tuning of BERT, BioBERT, and ClinicalBERT models as a binary mention–concept classifier, with a learned threshold for unlinkable mentions. The authors evaluate on ShARe/CLEF clinical notes, NCBI disease abstracts, and TAC2017ADR drug labels, comparing nine pre-trained models against BM25, a reimplemented CNN baseline, and published results from UWM, TaggerOne, Xu et al., and D'Souza and Ng. They report consistent state-of-the-art accuracy, with gains of 0.35%, 0.26%, and 1.17% on the three datasets. The central contribution is empirical: a demonstration that fine-tuned contextualized representations improve biomedical entity normalization.
Significance. If the reported gains are real, the paper provides a useful baseline for biomedical entity normalization and is among the first to apply BERT-family models to this task. The architecture is simple and clearly described, the coverage of nine pre-trained models is extensive, and the inclusion of a reimplemented CNN baseline on the same reconstructed knowledge bases is a strength. However, the incremental gains are very small on two of the three datasets, and the central claim rests on the trustworthiness of these small differences. The contribution is empirical rather than methodological, and no code or trained models are released.
major comments (4)
- [Parameters Settings; Table 4] The hyperparameter description states that batch size (16, 32) and training epochs (1 through 10) were tuned and "the model with the best performance" was saved, but it does not specify which data split was used for this selection. Since Tables 2 and 4 report test-set accuracies, the results as presented are consistent with model selection on the test set. With 20 hyperparameter configurations per model, test-set selection would inflate the reported accuracy and make the 0.26–1.17% gains uninterpretable. Please select epochs and batch size on a held-out development set and then report the test accuracy of the chosen configuration, or provide a correction for selection bias.
- [NCBI dataset; Table 3] The best NCBI model, BioBERTBase_Cased+PubMed+PMC, was pre-trained on PubMed abstracts and PMC full text, while the NCBI test set consists of 100 PubMed abstracts that are very likely contained in that pre-training corpus. Because the 0.26% gain over TaggerOne corresponds to roughly 2.5 mentions out of 960, even modest memorization of token-level patterns from the test abstracts could change the comparison. The paper neither reports whether the NCBI test abstracts overlap with BioBERT's pre-training data nor provides a control. Please quantify the overlap and rerun the NCBI comparison with a model that has not seen the test abstracts, or otherwise demonstrate that the result is unaffected.
- [Table 3; Comparisons with existing work] The reported differences on ShARe/CLEF and NCBI correspond to roughly 19 and 2.5 test mentions, respectively. No confidence intervals, bootstrap estimates, or significance tests are provided, and fine-tuning is stochastic, so the observed gaps could lie within run-to-run or sampling noise. Please report repeated runs or bootstrap confidence intervals (at least exact mention counts) and state whether the differences are statistically reliable.
- [Table 3; Comparisons with existing work] The comparison mixes problem settings and knowledge-base versions. TaggerOne is a joint NER+normalization model while the proposed system is normalization-only with gold mentions, and the UWM and D'Souza and Ng scores were obtained with their original knowledge bases while the proposed model uses a reconstructed SNOMED-CT subset. The paper acknowledges these differences but still claims "consistently outperformed." To support the claim, the comparisons should either use exactly the same evaluation protocol and knowledge bases, or report reimplementations (as was done for the CNN) for all baselines. At minimum, clarify whether the TaggerOne 88.80 result is normalization-only with gold mentions or an end-to-end NER+normalization result.
minor comments (5)
- [Parameters Settings] The phrase "saved the model with the best performance" should identify the evaluation metric and the split on which the choice was made; as written it is ambiguous and also hampers reproducibility.
- [Parameters Settings] The paper does not report the maximum sequence length, optimizer details, learning-rate schedule, or number of random seeds; these details are needed to reproduce the fine-tuning results.
- [Discussion] The statement that "BERT models with cased version were better than the uncased version in most cases" should be qualified with respect to TAC2017ADR, where the uncased variants outperform or tie the cased variants in Table 2.
- [Preprocessing; System Architecture] The preprocessing pipeline lowercases, stemms, and removes punctuation before fine-tuning, but BERT's WordPiece tokenizer has its own tokenization; please state explicitly whether the preprocessed mention is fed directly to BERT or re-tokenized, since this affects the validity of the cased-versus-uncased comparison.
- [Abstract and Notation] There are minor presentation issues: "we proposed" in the abstract should be "we propose," and the mathematical notation for the ranking score and candidate set is corrupted in the rendered text (e.g., 𝑠𝑐𝑜𝑟𝑒(𝑚,𝑐)=𝑠𝑜𝑓𝑡𝑚𝑎𝑥(𝐶𝑊P) and {𝑊Y}YZ[[\).
Circularity Check
No circularity: the reported results are measured accuracies on external benchmark test sets, with no load-bearing step that reduces to the paper's own inputs.
full rationale
This paper is an empirical evaluation, not a derivation: the central claim is that fine-tuning pre-trained BERT-family models improves entity-normalization accuracy on three benchmark datasets. Accuracy is computed on held-out test sets against external baselines, and the gains are measured, not fitted. The candidate ranking model is a standard sentence-pair classification fine-tuning procedure, and the NIL threshold is learned from a held-out development subset, which is a normal experimental design rather than a circular construction. The paper's use of self-citations is not load-bearing: the ClinicalBERT models cited from the same group are publicly released pre-trained models, the TAC2017ADR baseline from Xu et al. is a published challenge system, and none of these citations are invoked to forbid alternatives or to define the target result. The acknowledged limitations about not fully reconstructing prior KBs and the ambiguity of 'saved the model with the best performance' are methodological and reproducibility concerns about selection and comparability, not circularity, since the reported numbers still come from test-set evaluation and are not algebraically entailed by the hyperparameters or by any equation in the paper. No quoted step in the manuscript exhibits a reduction of a prediction to a fitted input or a self-citation chain that forces the conclusion, so no circularity is present.
Assumptions & free parameters
free parameters (5)
- number of training epochs =
1 to 10, selected by best performance
- batch size =
16 or 32, selected per dataset
- NIL threshold tau =
Learned from training data with a held-out development set
- learning rate =
2e-5
- BM25 top-k candidates =
10
assumptions (5)
- domain assumption Gold-standard annotations in the three datasets are correct and consistent.
- domain assumption The pre-trained BERT, BioBERT, and ClinicalBERT models provide useful representations for biomedical text.
- domain assumption BM25 candidate generation recalls the correct concept within the top 10 candidates for all linkable mentions.
- ad hoc to paper The reconstructed knowledge bases (UMLS subset, MEDIC, MedDRA) are close enough to those used in prior work for fair comparison.
- domain assumption Accuracy, the percentage of correctly normalized mentions, is an appropriate evaluation metric for this task.
Cite this review
Pith. "Pith review of BERT-based Ranking for Biomedical Entity Normalization." pith.science (2026). https://pith.science/paper/R4JRT6I4
@misc{pith2026190803548,
author = {Pith},
title = {Pith review of: BERT-based Ranking for Biomedical Entity Normalization},
year = {2026},
howpublished = {\url{https://pith.science/paper/R4JRT6I4}},
note = {Machine review of arXiv:1908.03548}
}
read the original abstract
Developing high-performance entity normalization algorithms that can alleviate the term variation problem is of great interest to the biomedical community. Although deep learning-based methods have been successfully applied to biomedical entity normalization, they often depend on traditional context-independent word embeddings. Bidirectional Encoder Representations from Transformers (BERT), BERT for Biomedical Text Mining (BioBERT) and BERT for Clinical Text Mining (ClinicalBERT) were recently introduced to pre-train contextualized word representation models using bidirectional Transformers, advancing the state-of-the-art for many natural language processing tasks. In this study, we proposed an entity normalization architecture by fine-tuning the pre-trained BERT / BioBERT / ClinicalBERT models and conducted extensive experiments to evaluate the effectiveness of the pre-trained models for biomedical entity normalization using three different types of datasets. Our experimental results show that the best fine-tuned models consistently outperformed previous methods and advanced the state-of-the-art for biomedical entity normalization, with up to 1.17% increase in accuracy.
Reference graph
Works this paper leans on
-
[1]
BERT-based Ranking for Biomedical Entity Normalization Zongcheng Ji, PhD1, Qiang Wei, MS1, Hua Xu, PhD1 1School of Biomedical Informatics, The University of Texas Health Science Center at Houston, Houston, TX, USA Abstract Developing high-performance entity normalization algorithms that can alleviate the term variation problem is of great interest to the ...
work page 2014
-
[2]
Entity Linking with a Knowledge Base: Issues, Techniques, and Solutions
Shen W, Wang J, Han J. Entity Linking with a Knowledge Base: Issues, Techniques, and Solutions. TKDE. 2014;99:1. doi:http://doi.ieeecomputersociety.org/10.1109/TKDE.2014.2327028
arXiv 2014
-
[3]
CNN-based ranking for biomedical entity normalization
Li H, Chen Q, Tang B, et al. CNN-based ranking for biomedical entity normalization. BMC Bioinformatics. 2017;18(11):385
work page 2017
-
[4]
Sieve-Based Entity Linking for the Biomedical Domain
D’Souza J, Ng V. Sieve-Based Entity Linking for the Biomedical Domain. In: ACL. ; 2015:297-302
work page 2015
-
[5]
Ghiasvand O, Kate RJ. UWM: Disorder Mention Extraction from Clinical Text Using CRFs and Normalization Using Learned Edit Distance Patterns. In: SemEval@COLING. ; 2014:828-832
work page 2014
-
[6]
Using rule-based natural language processing to improve disease normalization in biomedical text
Kang N, Singh B, Afzal Z, van Mulligen EM, Kors JA. Using rule-based natural language processing to improve disease normalization in biomedical text. JAMIA. 2012;20(5):876-881
work page 2012
-
[7]
DNorm: disease name normalization with pairwise learning to rank
Leaman R, Doǧan RI, Lu Z. DNorm: disease name normalization with pairwise learning to rank. Bioinformatics. 2013;29:2909-2917
work page 2013
-
[9]
The Unified Medical Language System (UMLS): integrating biomedical terminology
Bodenreider O. The Unified Medical Language System (UMLS): integrating biomedical terminology. Nucleic Acids Res. 2004;32(suppl_1):D267. doi:10.1093/nar/gkh061
Show all 32 references
-
[10]
SemEval-2014 Task 7: Analysis of Clinical Text
Pradhan S, Elhadad N, Chapman WW, Manandhar S, Savova G. SemEval-2014 Task 7: Analysis of Clinical Text. In: SemEval. ; 2014:54-62
2014
-
[11]
Large-scale linear ranksvm
Lee C-P, Lin C-J. Large-scale linear ranksvm. Neural Comput. 2014;26(4):781-817
2014
-
[12]
Overview of the TAC 2017 Adverse Reaction Extraction from Drug Labels Track
Roberts K, Demner-Fushman D, Tonning JM. Overview of the TAC 2017 Adverse Reaction Extraction from Drug Labels Track. In: TAC
2017
-
[14]
Distributed representations of words and phrases and their compositionality
Mikolov T, Sutskever I, Chen K, Corrado GS, Dean J. Distributed representations of words and phrases and their compositionality. In: Advances in Neural Information Processing Systems. ; 2013:3111-3119
2013
-
[15]
Deep Contextualized Word Representations
Peters ME, Neumann M, Iyyer M, et al. Deep Contextualized Word Representations. In: NAACL-HLT. ; 2018:2227-2237. https://aclanthology.info/papers/N18-1202/n18-1202
2018
-
[18]
BioBERT: a pre-trained biomedical language representation model for biomedical text mining
Lee J, Yoon W, Kim S, et al. BioBERT: a pre-trained biomedical language representation model for biomedical text mining. CoRR. 2019;abs/1901.0. http://arxiv.org/abs/1901.08746
2019 arXiv
-
[19]
Enhancing clinical concept extraction with contextual embeddings
Si Y, Wang J, Xu H, Roberts K. Enhancing clinical concept extraction with contextual embeddings. JAMIA. July 2019:ocz096. doi:10.1093/jamia/ocz096
2019 doi
-
[20]
ClinicalBERT: Modeling Clinical Notes and Predicting Hospital Readmission
Huang K, Altosaar J, Ranganath R. ClinicalBERT: Modeling Clinical Notes and Predicting Hospital Readmission. CoRR. 2019;abs/1904.0. http://arxiv.org/abs/1904.05342
2019 arXiv
-
[21]
Publicly Available Clinical BERT Embeddings
Alsentzer E, Murphy JR, Boag W, et al. Publicly Available Clinical BERT Embeddings. CoRR. 2019;abs/1904.0. http://arxiv.org/abs/1904.03323
2019 arXiv
-
[23]
Evaluating the state of the art in disorder recognition and normalization of the clinical narrative
Pradhan S, Elhadad N, South BR, et al. Evaluating the state of the art in disorder recognition and normalization of the clinical narrative. JAMIA. 2015;22(1):143-154. doi:10.1136/amiajnl-2013-002544
2015 doi
-
[24]
NCBI disease corpus: a resource for disease name recognition and concept normalization
Doǧan RI, Leaman R, Lu Z. NCBI disease corpus: a resource for disease name recognition and concept normalization. JBI. 2014;47:1-10
2014
-
[25]
MEDIC: a practical disease vocabulary used at the Comparative Toxicogenomics Database
Davis AP, Wiegers TC, Rosenstein MC, Mattingly CJ. MEDIC: a practical disease vocabulary used at the Comparative Toxicogenomics Database. Database. 2012;2012:bar065. doi:10.1093/database/bar065
2012 doi
-
[26]
Abbreviation definition identification based on automatic precision estimates
Sohn S, Comeau DC, Kim W, Wilbur WJ. Abbreviation definition identification based on automatic precision estimates. BMC Bioinformatics. 2008;9. doi:10.1186/1471-2105-9-402
2008 doi
-
[27]
A Simple Algorithm for Identifying Abbreviation Definitions in Biomedical Text
Schwartz AS, Hearst MA. A Simple Algorithm for Identifying Abbreviation Definitions in Biomedical Text. In: Proceedings of the 8th Pacific Symposium on Biocomputing. ; 2003:451-462
2003
-
[28]
CLAMP - a toolkit for efficiently building customized clinical natural language processing pipelines
Soysal E, Wang J, Jiang M, et al. CLAMP - a toolkit for efficiently building customized clinical natural language processing pipelines. JAMIA. 2017;25(3):331-336
2017
-
[29]
An Information Retrieval Approach to Short Text Conversation
Ji Z, Lu Z, Li H. An Information Retrieval Approach to Short Text Conversation. http://arxiv.org/abs/14086988
-
[30]
UTH-CCB: The Participation of the SemEval 2015 Challenge - Task
Xu J, Zhang Y, Wang J, et al. UTH-CCB: The Participation of the SemEval 2015 Challenge - Task
2015
-
[31]
2015:311-314
In: SemEval. ; 2015:311-314. http://www.aclweb.org/anthology/S15-2052
2015
-
[32]
UTH_CCB: a report for semeval 2014 - task 7 analysis of clinical text
Zhang Y, Wang J, Tang B, et al. UTH_CCB: a report for semeval 2014 - task 7 analysis of clinical text. In: SemEval. ; 2014:802
2014
-
[33]
Okapi at TREC-3
Robertson SE, Walker S, Jones S, Hancock-Beaulieu M, Gatford M. Okapi at TREC-3. In: Proceedings of TREC. ; 1995:109-126
1995
-
[34]
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin J, Chang M-W, Lee K, Toutanova K. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. CoRR. 2018;abs/1810.0. http://arxiv.org/abs/1810.04805
2018 arXiv
-
[35]
MIMIC-III, a freely accessible critical care database
Johnson AEW, Pollard TJ, Shen L, et al. MIMIC-III, a freely accessible critical care database. Sci data. 2016;3:160035
2016
-
[36]
TaggerOne: joint named entity recognition and normalization with semi-Markov Models
Leaman R, Lu Z. TaggerOne: joint named entity recognition and normalization with semi-Markov Models. Bioinformatics. 2016;32(18):2839-2846
2016
-
[38]
Joint Recognition and Linking of Fine-Grained Locations from Tweets
Ji Z, Sun A, Cong G, Han J. Joint Recognition and Linking of Fine-Grained Locations from Tweets. In: WWW. ; 2016:1271-1281. doi:10.1145/2872427.2883067
2016
Reviewed August 14, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.