RECORD
Deep learning with sentence embeddings pre-trained on biomedical corpora improves the performance of finding similar sentences in electronic medical records
Computation and Language · Submitted 6 Sep 2019
abstract compiled · 2026-09-23 · no language-model review
The authors' main claim Deep learning models with sentence embeddings pre-trained on biomedical corpora achieve the highest performance on the test set.
The compiled record
The paper's own sentences, arranged. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. Numbers in the tables are the authors', checked by code. No sentence in this record was written by a model.
Open the compiled record
The abstract, sentence by sentence
Capturing sentence semantics plays a vital role in a range of text mining applications. Despite continuous efforts on the development of related datasets and models in the general domain, both datasets and models are limited in biomedical and clinical domains. The BioCreative/OHNLP organizers have made the first attempt to annotate 1,068 sentence pairs from clinical notes and have called for a community effort to tackle the Semantic Textual Similarity (BioCreative/OHNLP STS) challenge. We developed models using traditional machine learning and deep learning approaches. For the post challenge, we focus on two models: the Random Forest and the Encoder Network. We applied sentence embeddings pre-trained on PubMed abstracts and MIMIC-III clinical notes and updated the Random Forest and the Encoder Network accordingly. The official results demonstrated our best submission was the ensemble of eight models. It achieved a Person correlation coefficient of 0.8328, the highest performance among 13 submissions from 4 teams. For the post challenge, the performance of both Random Forest and the Encoder Network was improved; in particular, the correlation of the Encoder Network was improved by ~13%. During the challenge task, no end-to-end deep learning models had better performance than machine learning models that take manually-crafted features. In contrast, with the sentence embeddings pre-trained on biomedical corpora, the Encoder Network now achieves a correlation of ~0.84, which is higher than the original best model. The ensembled model taking the improved versions of the Random Forest and Encoder Network as inputs further increased performance to 0.8528. Deep learning models with sentence embeddings pre-trained on biomedical corpora achieve the highest performance on the test set.
background contribution method result interpretation main claim
| # | sentence | role | the authors' own work | states the main conclusion | strength |
|---|---|---|---|---|---|
| s0 | Capturing sentence semantics plays a vital role in a range of text mining applications. | background 1.00 | 0.04 | 0.02 | asserted 1.00 |
| s1 | Despite continuous efforts on the development of related datasets and models in the general domain, both datasets and models are limited in biomedical and clinical domains. | gap 0.93 | 0.06 | 0.02 | asserted 1.00 |
| s2 | The BioCreative/OHNLP organizers have made the first attempt to annotate 1,068 sentence pairs from clinical notes and have called for a community effort to tackle the Semantic Textual Similarity (BioCreative/OHNLP STS) challenge. | background 0.86 | 0.08 | 0.02 | asserted 1.00 |
| s3 | We developed models using traditional machine learning and deep learning approaches. | method 0.89 | 0.93 | 0.02 | asserted 0.96 |
| s4 | For the post challenge, we focus on two models: the Random Forest and the Encoder Network. | method 0.96 | 0.80 | 0.02 | asserted 0.93 |
| s5 | We applied sentence embeddings pre-trained on PubMed abstracts and MIMIC-III clinical notes and updated the Random Forest and the Encoder Network accordingly. | method 0.97 | 0.94 | 0.05 | asserted 0.98 |
| s6 | The official results demonstrated our best submission was the ensemble of eight models. | result 0.99 | 0.91 | 0.06 | asserted 1.00 |
| s7 | It achieved a Person correlation coefficient of 0.8328, the highest performance among 13 submissions from 4 teams. | result 1.00 | 0.88 | 0.11 | asserted 1.00 |
| s8 | For the post challenge, the performance of both Random Forest and the Encoder Network was improved; in particular, the correlation of the Encoder Network was improved by ~13%. | result 1.00 | 0.94 | 0.14 | asserted 1.00 |
| s9 | During the challenge task, no end-to-end deep learning models had better performance than machine learning models that take manually-crafted features. | result 0.97 | 0.77 | 0.04 | asserted 1.00 |
| s10 | In contrast, with the sentence embeddings pre-trained on biomedical corpora, the Encoder Network now achieves a correlation of ~0.84, which is higher than the original best model. | result 0.96 | 0.92 | 0.35 | asserted 1.00 |
| s11 | The ensembled model taking the improved versions of the Random Forest and Encoder Network as inputs further increased performance to 0.8528. | result 1.00 | 0.95 | 0.14 | asserted 1.00 |
| s12 | Deep learning models with sentence embeddings pre-trained on biomedical corpora achieve the highest performance on the test set. | result 0.76 | 0.91 | 0.93 | asserted 1.00 |
What was examined, and what was not
- arXiv listing abstract v1
- full text
- figures and tables
- references and citing sentences
- relations to other papers
- P abstract
- S sections
- T body sentences
- C citations
- R cross-paper relations
The paper
Original abstract
Capturing sentence semantics plays a vital role in a range of text mining applications. Despite continuous efforts on the development of related datasets and models in the general domain, both datasets and models are limited in biomedical and clinical domains. The BioCreative/OHNLP organizers have made the first attempt to annotate 1,068 sentence pairs from clinical notes and have called for a community effort to tackle the Semantic Textual Similarity (BioCreative/OHNLP STS) challenge. We developed models using traditional machine learning and deep learning approaches. For the post challenge, we focus on two models: the Random Forest and the Encoder Network. We applied sentence embeddings pre-trained on PubMed abstracts and MIMIC-III clinical notes and updated the Random Forest and the Encoder Network accordingly. The official results demonstrated our best submission was the ensemble of eight models. It achieved a Person correlation coefficient of 0.8328, the highest performance among 13 submissions from 4 teams. For the post challenge, the performance of both Random Forest and the Encoder Network was improved; in particular, the correlation of the Encoder Network was improved by ~13%. During the challenge task, no end-to-end deep learning models had better performance than machine learning models that take manually-crafted features. In contrast, with the sentence embeddings pre-trained on biomedical corpora, the Encoder Network now achieves a correlation of ~0.84, which is higher than the original best model. The ensembled model taking the improved versions of the Random Forest and Encoder Network as inputs further increased performance to 0.8528. Deep learning models with sentence embeddings pre-trained on biomedical corpora achieve the highest performance on the test set.
Human review
Discussion
Know something about this paper? Leave a comment, or highlight any passage above to comment on it.