DisEmbed: Transforming Disease Understanding through Embeddings

Salman Faroz

REVIEW 4 major objections 5 minor 19 references

DisEmbed: Transforming Disease Understanding through Embeddings

T0 review · 4 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims that a 33M-parameter disease-focused embedding model, trained on synthetic symptom–disease pairs generated from ICD-10-CM names, outperforms larger medical and general embedding models on three disease-specific triplet…

desk verdict A useful synthetic disease dataset and fine-tuned model, but the SOTA claim is undercut by a missing train/test overlap check and an absent base-model baseline. read the letter →

arxiv 2412.15258 v1 pith:N4BACFZV submitted 2024-12-16 cs.CL cs.LG

Salman Faroz This is my paper · ORCID

classification cs.CLcs.LG

keywords diseaseembeddingssynthetictrainingdatatripletevaluationretrieval-augmentedgenerationICD-10-CMmedicalNLPsymptom–diseaseassociationsentence

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that a small, deliberately narrow embedding model can outperform much larger medical models on disease-specific tasks. The author builds DisEmbed-v1, a 33M-parameter model fine-tuned on synthetic disease descriptions, symptoms, and Q&A pairs generated from ICD-10-CM disease names with the disease name itself excluded from the generated text. On three disease-focused triplet benchmarks, the model reports higher accuracy than PubMedBERT, ClinicalBERT, Bio-ClinicalBERT, MedEmbed variants, and two large general-purpose embedders. A reader should care because the claim, if true, means compact models trained on targeted synthetic data can beat broad clinical models for disease retrieval and RAG, with far lower computational cost.

What carries the argument

The central mechanism is the synthetic training corpus combined with the deliberate hiding of disease names. GPT-4o-mini generates symptom descriptions and Q&A pairs from ICD-10-CM disease names, and the generated text never contains the disease name, so the model cannot shortcut to label matching. Training uses anchor–positive pairs with Multiple Negatives Ranking Loss, which pulls related disease–symptom contexts together and pushes them apart from in-batch negatives. Evaluation is the triplet condition $S(A,P) > S(A,N) + \text{margin}$, implemented through the sentence-transformers TripletEvaluator or a custom evaluator for models outside that library.

What would settle it

Compute lexical and semantic overlap between the synthetic training corpus and each of the three evaluation datasets; if any test triplet's disease–symptom pairs appear verbatim or near-verbatim in training, the reported advantage must be re-measured on deduplicated data. A stronger test is to retrain DisEmbed-v1 on a version of the corpus that excludes all diseases appearing in the three benchmarks, then check whether the accuracy gap over the baseline models persists.

Watch

Extended reading notes

Core claim

DisEmbed-v1 is built by fine-tuning BAAI/bge-small-en-v1.5 with Multiple Negatives Ranking Loss on anchor–positive pairs drawn from a synthetic corpus. The corpus is created by passing over 70,000 ICD-10-CM disease names through GPT-4o-mini to generate symptom lists, descriptions, and Q&A pairs that never name the disease, which the author argues forces the model to learn disease concepts from clinical content rather than from surface labels. On the triplet evaluation, DisEmbed-v1 reports 94.5% accuracy on the Disease Database, 91.6% on the Medical Diagnosis Dialogue dataset, and 93.7% on CoD-PatientSymDisease, the highest scores among all compared models despite having only 33M parameters. A cosine-similarity probe shows that a neuropathy-like symptom description scores 0.6333 against neuropathy but only 0.1062 against epilepsy syndrome for DisEmbed-v1, while models such as BioBERT assign nearly equal similarity to both, which the paper reads as evidence of disease-sensitive representations rather than generic medical similarity.

Load-bearing premise

The load-bearing assumption is that the synthetic training data generated from ICD-10-CM names does not overlap with the three evaluation datasets; the paper reports no deduplication or overlap analysis, so if GPT-4o-mini reproduced or paraphrased any test disease–symptom associations, the benchmark accuracy would reflect memorization rather than generalization.

Editorial extensions

If this is right

Disease-focused retrieval-augmented generation can run on a 33M-parameter retriever that outperforms larger clinical embedders on disease ranking tasks.
Synthetic data generated from a structured terminology such as ICD-10-CM can substitute for large private clinical corpora when building narrow-domain medical embedders.
The triplet-accuracy gains translate into more reliable symptom-to-disease ranking, which is the core operation in clinical decision support and diagnosis aids.
Compact disease-specific embedders can be embedded in clinical workflows where full-size clinical BERT models are impractical.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If the training data overlaps the test sets, the reported accuracy gap would shrink or vanish; the absence of a deduplication check in the paper makes this the most direct threat to the main claim.
The same 'hide the label' synthetic-data recipe could be tested on other narrow medical domains, such as medications or procedures, to see whether compact specialist embedders generalize beyond diseases.
Triplet accuracy at a single margin is a coarse measure; retrieval recall@k or graded similarity ranking would test whether the reported separation holds when many diseases compete in a large embedding database.
The comparison is against models that were not trained on these disease datasets, so a cleaner experiment would fine-tune the same base model on the same synthetic corpus with different objectives to isolate the contribution of the data-generation design.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Colleague,

Here's my read of DisEmbed. The genuinely new thing is a disease-focused synthetic training set: over 70k ICD-10-CM disease names run through GPT-4o-mini to generate symptom lists, descriptions, and Q&A pairs, then used to fine-tune BAAI/bge-small-en-v1.5 (33M params) with Multiple Negatives Ranking Loss. The model and data are on Hugging Face, so the work is reproducible. On three disease-specific triplet sets, DisEmbed beats several larger clinical models by 3–6 points. That is a useful resource for disease retrieval in RAG and clinical decision support, though not a conceptual breakthrough.

The paper is honest about some things: it acknowledges the synthetic data may carry biases, and it doesn't oversell beyond disease-specific tasks. The evaluation is simple and mostly appropriate for what it claims. But there are real soft spots.

The biggest issue: no overlap check between the synthetic training set and the three test sets. Since GPT-4o-mini was pretrained on the open web, it could easily have absorbed Disease Database or CoD-PatientSymDisease. If paraphrases of those symptom–disease associations show up in training, Table 1 measures memorization, not disease understanding. The paper reports no deduplication or near-duplicate removal. That is a load-bearing gap, not a minor one. The author's own limitation sentence covers noise, not contamination.

Second, the base model BAAI/bge-small-en-v1.5 is missing from Table 1. Without it, you can't tell whether the gains come from fine-tuning or from the base model's own quality. That is a straightforward fix and should have been there.

Third, no error bars, no repeated seeds, and two baselines (ClinicalBERT, BioBERT) are evaluated with a nonstandard mean-pooling scheme that likely hurts them. That biases the comparison and makes the 3–6 point margins look fragile. The cosine example in Table 2 is anecdotal; fine as an illustration, but it doesn't carry weight.

Is the central argument sound? The model may well be useful, but the evidence for SOTA is not yet convincing. Still, these are fixable: run the overlap analysis, add the base model, add seeds, report variance, and use standard pooling. The author has shipped the artifacts, so any reviewer can check.

Who is this for? People working on medical retrieval or RAG who might want a small disease-specific embedder. It deserves a serious referee, but not acceptance as-is. I'd send it to review with major-revision expectations.

Referee Report

4 major / 5 minor

Summary. The paper introduces DisEmbed-v1, a 33M-parameter disease-focused embedding model fine-tuned from BAAI/bge-small-en-v1.5 on a synthetic dataset generated by GPT-4o-mini from ICD-10-CM disease names. The synthetic data consists of disease descriptions, symptom lists, and Q&A pairs, and the model is trained with Multiple Negatives Ranking Loss. The central claim is that DisEmbed-v1 achieves state-of-the-art results on three disease-related triplet benchmarks (Disease Database, Medical Diagnosis Dialogue, and CoD-PatientSymDisease), outperforming larger medical and general-purpose embedding models. The paper also includes a single cosine-similarity anecdote comparing neuropathy versus epilepsy syndrome and discusses implications for retrieval-augmented generation.

Significance. If the central claim held, a compact 33M disease-specific embedding model with public weights and a public synthetic training set would be a practically useful contribution for medical retrieval and RAG applications. The availability of the model and dataset is a genuine strength, as is the focus on a specific and underexplored subdomain. However, the empirical evidence as presented does not establish the claimed state-of-the-art performance: the base model is absent from the comparison, no statistical reliability information is provided, the triplet construction is unspecified, and the synthetic training data may overlap with the evaluation data. These issues are load-bearing because the headline results are a set of small accuracy differences in Table 1.

major comments (4)

[§3.2, §4.2, Table 1] The base model BAAI/bge-small-en-v1.5, which is the stated initialization for DisEmbed-v1, is not evaluated in Table 1. Without that baseline, the table cannot show whether fine-tuning on the synthetic disease data improves or degrades the base model; the reported 94.5/91.6/93.7 could reflect the base model's own quality rather than a benefit of the proposed training data. This omission directly affects the interpretation of the 'state-of-the-art' claim.
[§3.1 vs. §4.2.1] The training set is generated by GPT-4o-mini from ICD-10-CM disease names, while two of the test sets (Disease Database and CoD-PatientSymDisease) are also collections of symptom-disease associations. The paper reports no overlap analysis, no deduplication, and no removal of near-duplicate test triplets from the training set. If GPT-4o-mini reproduced or paraphrased test associations during synthetic data generation, the accuracies in Table 1 reflect memorization rather than generalization. This is the central validity threat to the paper's main claim.
[§4.2] The triplet construction is not described. The paper does not report how many triplets were derived from each dataset, how anchors, positives, and negatives were sampled, whether negatives are random or hard examples, or whether the exact same triplets were used for every model. Triplet accuracy is highly sensitive to negative sampling difficulty, so without this information the numbers in Table 1 are not reproducible and not comparable across models.
[§4.1.1, Table 1] ClinicalBERT and BioBERT are evaluated with a custom mean-pooling evaluator using the formula E(x) = (1/n) Σ x_i, while other models are evaluated through the sentence-transformers TripletEvaluator. The paper gives no evidence that these pooling strategies are equivalent for the compared BERT variants, and no error bars, confidence intervals, or repeated training runs are reported for any table entry. Since some accuracy gaps over the closest baselines are only 1–3 percentage points (e.g., 91.6 vs. 89.9 on MDD and 93.7 vs. 92.6 on CoD-PatientSymDisease), pooling mismatch or sampling noise could change the ranking.

minor comments (5)

[§1] The example pairing 'brain surgery' and 'Parkinson's disease' is misleading, because deep brain stimulation is a common surgical treatment for Parkinson's disease; a high cosine similarity between these terms may reflect a genuine relatedness rather than a model deficiency.
[§4.2.3] The Limitations subsection acknowledges the model's narrow medical scope but does not mention the absence of the base-model baseline or the potential training/test overlap, both of which are more immediate threats to the reported results.
[§3.2] The sentence 'the model was not trained with a triplet approach' is confusing, because Multiple Negatives Ranking Loss is a triplet-style ranking objective; please clarify the intended distinction.
[§4.1.1] The custom evaluator's term 'token embedding' is undefined; please specify whether x_i refers to input word embeddings, last hidden states, or another representation, and describe how special tokens and subword tokenization are handled.
[References] Several references are incomplete or inconsistently formatted, such as [1] lacking a venue, [5] lacking author names, and [13] lacking a URL or version identifier.

Circularity Check

0 steps flagged · score 0.0 of 10

No demonstrated circularity: the training and evaluation data are separate sources; a data-contamination risk is noted but not established.

full rationale

The paper's derivation chain is not circular. DisEmbed is trained with Multiple Negatives Ranking Loss on a synthetic anchor-positive dataset generated by GPT-4o-mini from ICD-10-CM disease names, and evaluated on three external disease-symptom datasets via triplet accuracy. Training and evaluation both use disease-symptom association structure, but that is the normal supervised-learning setup, not an equivalence by construction: the reported test accuracies are not defined in terms of the training set, no test triplet is shown to be a training item, and no parameter is fitted to the evaluation benchmarks. No fitted constant is renamed as a prediction, and no load-bearing self-citation or imported uniqueness theorem is used to force the model choice. The only substantive concern is data contamination: Section 3.1 does not report deduplication of the synthetic training set against Disease Database, MDD, or CoD-PatientSymDisease, and GPT-4o-mini's pretraining corpus may contain these public datasets. That is a validity risk that the paper itself gestures at in its caution about synthetic-data biases, but the text provides no specific reduction showing a test association appears in training, so it does not meet the standard for a demonstrated circular step. Score 0.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The paper introduces no new free parameters, no new axioms beyond standard machine learning assumptions, and no invented entities. The main assumptions concern the validity of the data generation and evaluation methodology.

assumptions (3)

domain assumption GPT-4o-mini generates accurate and representative disease descriptions and symptom lists.
The entire synthetic training set is produced by GPT-4o-mini (Section 3.1); if the generator introduces factual errors or biases, the model learns those errors.
domain assumption Triplet accuracy on the three chosen datasets measures disease understanding.
The paper uses triplet accuracy as the central metric and extrapolates to RAG performance without a direct downstream evaluation (Sections 4.1 and 5.2).
domain assumption The negative samples in the evaluation triplets are easy enough to distinguish, and the positives are truly related.
The triplet construction procedure is not described; if negatives are weak or duplicates, accuracy is inflated (Section 4.1).

how reviews work

0 comments

Cite this review

Pith. "Pith review of DisEmbed: Transforming Disease Understanding through Embeddings." pith.science (2026). https://pith.science/paper/N4BACFZV

@misc{pith2026241215258,
  author       = {Pith},
  title        = {Pith review of: DisEmbed: Transforming Disease Understanding through Embeddings},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/N4BACFZV}},
  note         = {Machine review of arXiv:2412.15258}
}

read the original abstract

The medical domain is vast and diverse, with many existing embedding models focused on general healthcare applications. However, these models often struggle to capture a deep understanding of diseases due to their broad generalization across the entire medical field. To address this gap, I present DisEmbed, a disease-focused embedding model. DisEmbed is trained on a synthetic dataset specifically curated to include disease descriptions, symptoms, and disease-related Q\&A pairs, making it uniquely suited for disease-related tasks. For evaluation, I benchmarked DisEmbed against existing medical models using disease-specific datasets and the triplet evaluation method. My results demonstrate that DisEmbed outperforms other models, particularly in identifying disease-related contexts and distinguishing between similar diseases. This makes DisEmbed highly valuable for disease-specific use cases, including retrieval-augmented generation (RAG) tasks, where its performance is particularly robust.

Figures

Figures reproduced from arXiv: 2412.15258 by the authors.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

19 extracted references · 15 canonical work pages

[1]

Generalist embedding models are better at short-context clinical semantic search than specialized embedding models, 2024

Nicolas Excoffier and Benjamin Roehr. Generalist embedding models are better at short-context clinical semantic search than specialized embedding models, 2024

work page 2024
[2]

International classification of diseases, tenth revision, clinical modification (icd-10-cm)

ICD10Data.com. International classification of diseases, tenth revision, clinical modification (icd-10-cm)

work page
[3]

Neuml/pubmedbert-base-embeddings

neuML. Neuml/pubmedbert-base-embeddings

work page
[4]

Medcoder: A generative ai assistant for medical coding, 2023

Jun Li, Yu Sun, Yiming Peng, and Zhiyuan Liu. Medcoder: A generative ai assistant for medical coding, 2023

work page 2023
[5]

Medembed: Pre-trained medical embeddings

Abhinand et al. Medembed: Pre-trained medical embeddings

work page
[6]

Scibert: A pretrained language model for scientific text

Iz Beltagy, Kyle Lo, and Arman Cohan. Scibert: A pretrained language model for scientific text. arXiv preprint arXiv:1903.10676, 2019

arXiv 1903
[7]

Transfer learning in biomedical natural language processing: An evaluation of bert and elmo on ten benchmarking datasets

Yifan Peng, Shankai Yan, and Zhiyong Lu. Transfer learning in biomedical natural language processing: An evaluation of bert and elmo on ten benchmarking datasets. arXiv preprint arXiv:1906.05474, 2019

arXiv 1906
[8]

Clinicalbert: Modeling clinical notes and predicting hospital readmission

Kexin Huang, Jaan Altosaar, and Rajesh Ranganath. Clinicalbert: Modeling clinical notes and predicting hospital readmission. arXiv preprint arXiv:1904.05342, 2019

arXiv 1904

Show all 19 references

[9]

Publicly available clinical bert embeddings

Emily Alsentzer, John R Murphy, Willie Boag, Wei-Hung Weng, Di Jin, Tristan Naumann, and Matthew McDermott. Publicly available clinical bert embeddings. arXiv preprint arXiv:1904.03323, 2019

1904 arXiv
[10]

Efficient natural language response suggestion for smart reply, 2017

Matthew Henderson, Rami Al-Rfou, Brian Strope, Yun hsuan Sung, Laszlo Lukacs, Ruiqi Guo, Sanjiv Kumar, Balint Miklos, and Ray Kurzweil. Efficient natural language response suggestion for smart reply, 2017

2017
[11]

Disease database

Benyou W ANG. Disease database
[12]

Medical diagnosis dialogue workshop @ iclr 2021

ICLR. Medical diagnosis dialogue workshop @ iclr 2021. In ICLR 2021 Workshop, 2021

2021
[13]

Cod-patientsymdisease

Benyou W ANG. Cod-patientsymdisease
[14]

Task-oriented dialogue system for automatic diagnosis, 2018

Zhongyu Wei Qianlong Liu Baolin Peng Huaixiao Tou Ting Chen Xuanjing Huang Kam fai Wong Xiangying Dai. Task-oriented dialogue system for automatic diagnosis, 2018

2018
[15]

Clinicalbert

medicalai. Clinicalbert
[16]

Bio-clinicalbert

Emily Alsentzer. Bio-clinicalbert
[17]

Snowflake arctic

Snowflake. Snowflake arctic. 7
[18]

Baai/bge-large-en

Beijing Academy of Artificial Intelligence. Baai/bge-large-en
[19]

biobert-v1.1

Data Mining and Information Systems Lab. biobert-v1.1. 8

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.

[1] [1]

Generalist embedding models are better at short-context clinical semantic search than specialized embedding models, 2024

Nicolas Excoffier and Benjamin Roehr. Generalist embedding models are better at short-context clinical semantic search than specialized embedding models, 2024

work page 2024

[2] [2]

International classification of diseases, tenth revision, clinical modification (icd-10-cm)

ICD10Data.com. International classification of diseases, tenth revision, clinical modification (icd-10-cm)

work page

[3] [3]

Neuml/pubmedbert-base-embeddings

neuML. Neuml/pubmedbert-base-embeddings

work page

[4] [4]

Medcoder: A generative ai assistant for medical coding, 2023

Jun Li, Yu Sun, Yiming Peng, and Zhiyuan Liu. Medcoder: A generative ai assistant for medical coding, 2023

work page 2023

[5] [5]

Medembed: Pre-trained medical embeddings

Abhinand et al. Medembed: Pre-trained medical embeddings

work page

[6] [6]

Scibert: A pretrained language model for scientific text

Iz Beltagy, Kyle Lo, and Arman Cohan. Scibert: A pretrained language model for scientific text. arXiv preprint arXiv:1903.10676, 2019

arXiv 1903

[7] [7]

Transfer learning in biomedical natural language processing: An evaluation of bert and elmo on ten benchmarking datasets

Yifan Peng, Shankai Yan, and Zhiyong Lu. Transfer learning in biomedical natural language processing: An evaluation of bert and elmo on ten benchmarking datasets. arXiv preprint arXiv:1906.05474, 2019

arXiv 1906

[8] [8]

Clinicalbert: Modeling clinical notes and predicting hospital readmission

Kexin Huang, Jaan Altosaar, and Rajesh Ranganath. Clinicalbert: Modeling clinical notes and predicting hospital readmission. arXiv preprint arXiv:1904.05342, 2019

arXiv 1904