{"id":"bcf133e1-ea14-4d44-8ff4-28984d56fa51","arxiv_id":"2412.15258","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"DisEmbed, a 33M-parameter disease-specific embedding model trained on synthetic symptom-disease data, claims state-of-the-art triplet accuracy on three disease benchmarks.","lead":"DisEmbed is a disease-focused text embedding model, fine-tuned on a synthetic dataset of disease descriptions and symptoms generated by GPT-4o-mini. The paper reports it outperforms larger medical and general embedding models on disease-specific triplet benchmarks, which could help retrieval systems better distinguish between related and unrelated diseases.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The synthetic training set is not shown to be disjoint from the three test datasets; if GPT-4o-mini reproduced test symptom–disease associations, Table 1 reflects memorization rather than generalization.","rationale":"The reader's weakest assumption and my load-bearing concern coincide: the absence of any demonstrated disjointness between the GPT-4o-mini-generated training data and the evaluation datasets. This is the right focal point because the paper's headline result is a set of triplet accuracies that are only meaningful if the model is being asked to generalize to unseen disease–symptom relationships. If the synthetic generator reproduced test associations, the reported margins are an artifact of data overlap. The paper's own Sections 3.1 and 7 make the check feasible, since both the training data and the evaluation datasets are publicly linked, so the proposed overlap analysis can be run directly. Secondary issues—missing error bars, no base-model baseline, and the nonstandard mean-token pooling for ClinicalBERT and BioBERT—would further weaken the comparison, but they are not the single most load-bearing problem because the leakage concern undermines the model's own score even before comparing baselines. I therefore keep the reader's REJECT verdict unchanged: the paper should not be accepted in its current form, though a clean overlap analysis could alter that assessment. I see no reason to manufacture a different objection; the leakage risk is real, specific, and testable.","tokens_in":5939,"tokens_out":3839,"duration_ms":39184,"concrete_test":"Download the Section 7 training dataset (DisEmbed-Symptom-Disease-v1) and the three Section 4.2.1 test datasets. Normalize all text (lowercase, strip punctuation, collapse whitespace). For every test triplet (A, P, N), test whether A or P appears verbatim or as a near-duplicate (e.g., token-set Jaccard similarity at least 0.8) of any training sample, and report the overlap rate per dataset. Then re-run the Table 1 triplet evaluation using only test triplets with no overlapping training text. If DisEmbed-v1's accuracy on this disjoint subset is no longer above every baseline—especially MedEmbed-Small-v0.1 and BAAI-bge-large-en-v1.5—by a comparable margin, the state-of-the-art claim is unsupported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—Table 1's roughly 3–6 point advantage over same-size MedEmbed-Small-v0.1 and over larger clinical models—requires that DisEmbed-v1's training signal measures disease understanding, not recall of evaluation content. Section 3.1 builds the training set by asking GPT-4o-mini to generate symptoms and descriptions for ICD-10-CM disease names, while Section 4.2.1 evaluates on Disease Database and CoD-PatientSymDisease, which are also symptom-to-disease association collections. Because GPT-4o-mini was pretrained on the open web, it is entirely plausible that the generated 'synthetic' symptom lists paraphrase or duplicate the exact associations used in those test sets. The paper reports no overlap analysis, no deduplication, and no removal of near-duplicate test triplets from the training set. Without such evidence, DisEmbed-v1's higher triplet accuracy can be explained by memorized test associations rather than by learning disease structure. This is the load-bearing weakness because it attacks the validity of DisEmbed-v1's own score, not merely the fairness of the baseline comparisons.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces DisEmbed-v1, a 33M-parameter disease-focused embedding model fine-tuned from BAAI/bge-small-en-v1.5 on a synthetic dataset generated by GPT-4o-mini from ICD-10-CM disease names. The synthetic data consists of disease descriptions, symptom lists, and Q&A pairs, and the model is trained with Multiple Negatives Ranking Loss. The central claim is that DisEmbed-v1 achieves state-of-the-art results on three disease-related triplet benchmarks (Disease Database, Medical Diagnosis Dialogue, and CoD-PatientSymDisease), outperforming larger medical and general-purpose embedding models. The paper also includes a single cosine-similarity anecdote comparing neuropathy versus epilepsy syndrome and discusses implications for retrieval-augmented generation.","tokens_in":6158,"tokens_out":5221,"duration_ms":51647,"significance":"If the central claim held, a compact 33M disease-specific embedding model with public weights and a public synthetic training set would be a practically useful contribution for medical retrieval and RAG applications. The availability of the model and dataset is a genuine strength, as is the focus on a specific and underexplored subdomain. However, the empirical evidence as presented does not establish the claimed state-of-the-art performance: the base model is absent from the comparison, no statistical reliability information is provided, the triplet construction is unspecified, and the synthetic training data may overlap with the evaluation data. These issues are load-bearing because the headline results are a set of small accuracy differences in Table 1.","major_comments":[{"comment":"The base model BAAI/bge-small-en-v1.5, which is the stated initialization for DisEmbed-v1, is not evaluated in Table 1. Without that baseline, the table cannot show whether fine-tuning on the synthetic disease data improves or degrades the base model; the reported 94.5/91.6/93.7 could reflect the base model's own quality rather than a benefit of the proposed training data. This omission directly affects the interpretation of the 'state-of-the-art' claim.","section":"§3.2, §4.2, Table 1"},{"comment":"The training set is generated by GPT-4o-mini from ICD-10-CM disease names, while two of the test sets (Disease Database and CoD-PatientSymDisease) are also collections of symptom-disease associations. The paper reports no overlap analysis, no deduplication, and no removal of near-duplicate test triplets from the training set. If GPT-4o-mini reproduced or paraphrased test associations during synthetic data generation, the accuracies in Table 1 reflect memorization rather than generalization. This is the central validity threat to the paper's main claim.","section":"§3.1 vs. §4.2.1"},{"comment":"The triplet construction is not described. The paper does not report how many triplets were derived from each dataset, how anchors, positives, and negatives were sampled, whether negatives are random or hard examples, or whether the exact same triplets were used for every model. Triplet accuracy is highly sensitive to negative sampling difficulty, so without this information the numbers in Table 1 are not reproducible and not comparable across models.","section":"§4.2"},{"comment":"ClinicalBERT and BioBERT are evaluated with a custom mean-pooling evaluator using the formula E(x) = (1/n) Σ x_i, while other models are evaluated through the sentence-transformers TripletEvaluator. The paper gives no evidence that these pooling strategies are equivalent for the compared BERT variants, and no error bars, confidence intervals, or repeated training runs are reported for any table entry. Since some accuracy gaps over the closest baselines are only 1–3 percentage points (e.g., 91.6 vs. 89.9 on MDD and 93.7 vs. 92.6 on CoD-PatientSymDisease), pooling mismatch or sampling noise could change the ranking.","section":"§4.1.1, Table 1"}],"minor_comments":[{"comment":"The example pairing 'brain surgery' and 'Parkinson's disease' is misleading, because deep brain stimulation is a common surgical treatment for Parkinson's disease; a high cosine similarity between these terms may reflect a genuine relatedness rather than a model deficiency.","section":"§1"},{"comment":"The Limitations subsection acknowledges the model's narrow medical scope but does not mention the absence of the base-model baseline or the potential training/test overlap, both of which are more immediate threats to the reported results.","section":"§4.2.3"},{"comment":"The sentence 'the model was not trained with a triplet approach' is confusing, because Multiple Negatives Ranking Loss is a triplet-style ranking objective; please clarify the intended distinction.","section":"§3.2"},{"comment":"The custom evaluator's term 'token embedding' is undefined; please specify whether x_i refers to input word embeddings, last hidden states, or another representation, and describe how special tokens and subword tokenization are handled.","section":"§4.1.1"},{"comment":"Several references are incomplete or inconsistently formatted, such as [1] lacking a venue, [5] lacking author names, and [13] lacking a URL or version identifier.","section":"References"}],"recommendation":"reject","confidential_remarks":"The paper has useful public artifacts, but the evaluation design is not yet at the level needed to support the headline claim. The most serious issue is potential contamination between GPT-4o-mini-generated training data and the three test sets; an overlap analysis is essential. If the author can add the base-model baseline, describe triplet construction, provide statistical reliability measures, and either demonstrate or mitigate overlap, a revised submission could be reconsidered. I see no evidence of misconduct, but the current manuscript does not establish generalization."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nHere's my read of DisEmbed. The genuinely new thing is a disease-focused synthetic training set: over 70k ICD-10-CM disease names run through GPT-4o-mini to generate symptom lists, descriptions, and Q&A pairs, then used to fine-tune BAAI/bge-small-en-v1.5 (33M params) with Multiple Negatives Ranking Loss. The model and data are on Hugging Face, so the work is reproducible. On three disease-specific triplet sets, DisEmbed beats several larger clinical models by 3–6 points. That is a useful resource for disease retrieval in RAG and clinical decision support, though not a conceptual breakthrough.\n\nThe paper is honest about some things: it acknowledges the synthetic data may carry biases, and it doesn't oversell beyond disease-specific tasks. The evaluation is simple and mostly appropriate for what it claims. But there are real soft spots.\n\nThe biggest issue: no overlap check between the synthetic training set and the three test sets. Since GPT-4o-mini was pretrained on the open web, it could easily have absorbed Disease Database or CoD-PatientSymDisease. If paraphrases of those symptom–disease associations show up in training, Table 1 measures memorization, not disease understanding. The paper reports no deduplication or near-duplicate removal. That is a load-bearing gap, not a minor one. The author's own limitation sentence covers noise, not contamination.\n\nSecond, the base model BAAI/bge-small-en-v1.5 is missing from Table 1. Without it, you can't tell whether the gains come from fine-tuning or from the base model's own quality. That is a straightforward fix and should have been there.\n\nThird, no error bars, no repeated seeds, and two baselines (ClinicalBERT, BioBERT) are evaluated with a nonstandard mean-pooling scheme that likely hurts them. That biases the comparison and makes the 3–6 point margins look fragile. The cosine example in Table 2 is anecdotal; fine as an illustration, but it doesn't carry weight.\n\nIs the central argument sound? The model may well be useful, but the evidence for SOTA is not yet convincing. Still, these are fixable: run the overlap analysis, add the base model, add seeds, report variance, and use standard pooling. The author has shipped the artifacts, so any reviewer can check.\n\nWho is this for? People working on medical retrieval or RAG who might want a small disease-specific embedder. It deserves a serious referee, but not acceptance as-is. I'd send it to review with major-revision expectations.","headline":"A useful synthetic disease dataset and fine-tuned model, but the SOTA claim is undercut by a missing train/test overlap check and an absent base-model baseline.","tokens_in":6624,"tokens_out":2643,"would_cite":false,"duration_ms":25850,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that a 33M-parameter disease-focused embedding model, trained on synthetic symptom–disease pairs generated from ICD-10-CM names, outperforms larger medical and general embedding models on three disease-specific triplet…","keywords":["disease embeddings","synthetic training data","triplet evaluation","retrieval-augmented generation","ICD-10-CM","medical NLP","symptom–disease association","sentence embeddings"],"falsifier":"Compute lexical and semantic overlap between the synthetic training corpus and each of the three evaluation datasets; if any test triplet's disease–symptom pairs appear verbatim or near-verbatim in training, the reported advantage must be re-measured on deduplicated data. A stronger test is to retrain DisEmbed-v1 on a version of the corpus that excludes all diseases appearing in the three benchmarks, then check whether the accuracy gap over the baseline models persists.","tokens_in":5755,"feed_emoji":"🩺","tokens_out":8528,"duration_ms":66897,"temperature":0.7,"pith_summary":"This paper tries to establish that a small, deliberately narrow embedding model can outperform much larger medical models on disease-specific tasks. The author builds DisEmbed-v1, a 33M-parameter model fine-tuned on synthetic disease descriptions, symptoms, and Q&A pairs generated from ICD-10-CM disease names with the disease name itself excluded from the generated text. On three disease-focused triplet benchmarks, the model reports higher accuracy than PubMedBERT, ClinicalBERT, Bio-ClinicalBERT, MedEmbed variants, and two large general-purpose embedders. A reader should care because the claim, if true, means compact models trained on targeted synthetic data can beat broad clinical models for disease retrieval and RAG, with far lower computational cost.","feed_headline":"33M-parameter disease embedding beats larger medical models","feed_subtitle":"Trained on synthetic symptom data, DisEmbed-v1 beats PubMedBERT and larger rivals on three disease benchmarks.","key_machinery":"The central mechanism is the synthetic training corpus combined with the deliberate hiding of disease names. GPT-4o-mini generates symptom descriptions and Q&A pairs from ICD-10-CM disease names, and the generated text never contains the disease name, so the model cannot shortcut to label matching. Training uses anchor–positive pairs with Multiple Negatives Ranking Loss, which pulls related disease–symptom contexts together and pushes them apart from in-batch negatives. Evaluation is the triplet condition $S(A,P) > S(A,N) + \\text{margin}$, implemented through the sentence-transformers TripletEvaluator or a custom evaluator for models outside that library.","core_discovery":"DisEmbed-v1 is built by fine-tuning BAAI/bge-small-en-v1.5 with Multiple Negatives Ranking Loss on anchor–positive pairs drawn from a synthetic corpus. The corpus is created by passing over 70,000 ICD-10-CM disease names through GPT-4o-mini to generate symptom lists, descriptions, and Q&A pairs that never name the disease, which the author argues forces the model to learn disease concepts from clinical content rather than from surface labels. On the triplet evaluation, DisEmbed-v1 reports 94.5% accuracy on the Disease Database, 91.6% on the Medical Diagnosis Dialogue dataset, and 93.7% on CoD-PatientSymDisease, the highest scores among all compared models despite having only 33M parameters. A cosine-similarity probe shows that a neuropathy-like symptom description scores 0.6333 against neuropathy but only 0.1062 against epilepsy syndrome for DisEmbed-v1, while models such as BioBERT assign nearly equal similarity to both, which the paper reads as evidence of disease-sensitive representations rather than generic medical similarity.","pith_inferences":["If the training data overlaps the test sets, the reported accuracy gap would shrink or vanish; the absence of a deduplication check in the paper makes this the most direct threat to the main claim.","The same 'hide the label' synthetic-data recipe could be tested on other narrow medical domains, such as medications or procedures, to see whether compact specialist embedders generalize beyond diseases.","Triplet accuracy at a single margin is a coarse measure; retrieval recall@k or graded similarity ranking would test whether the reported separation holds when many diseases compete in a large embedding database.","The comparison is against models that were not trained on these disease datasets, so a cleaner experiment would fine-tune the same base model on the same synthetic corpus with different objectives to isolate the contribution of the data-generation design."],"forward_implications":["Disease-focused retrieval-augmented generation can run on a 33M-parameter retriever that outperforms larger clinical embedders on disease ranking tasks.","Synthetic data generated from a structured terminology such as ICD-10-CM can substitute for large private clinical corpora when building narrow-domain medical embedders.","The triplet-accuracy gains translate into more reliable symptom-to-disease ranking, which is the core operation in clinical decision support and diagnosis aids.","Compact disease-specific embedders can be embedded in clinical workflows where full-size clinical BERT models are impractical."],"supporting_citations":[{"why":"The prior study whose ICD-10-based evaluation the paper argues is inadequate for measuring disease semantics.","marker":"[1]"},{"why":"Supplies the 70,000+ ICD-10-CM disease names used to generate the synthetic training corpus.","marker":"[2]"},{"why":"PubMedBERT, a larger medical embedding baseline that DisEmbed-v1 claims to outperform in Table 1.","marker":"[3]"},{"why":"MedEmbed, the medical embedding model family used as a baseline in Table 1.","marker":"[5]"},{"why":"Defines the triplet evaluation condition used for all benchmark accuracy scores.","marker":"[10]"},{"why":"Disease Database, the first of the three disease–symptom benchmarks in Table 1.","marker":"[11]"},{"why":"Medical Diagnosis Dialogue dataset, the second benchmark in Table 1.","marker":"[12]"},{"why":"CoD-PatientSymDisease, the third benchmark in Table 1.","marker":"[13]"}],"fun_headline_variants":["DisEmbed beats larger medical models on disease tasks","33M-parameter disease model outperforms medical giants","Synthetic data helps small embedding win disease benchmarks","Disease-focused embedding tops PubMedBERT and rivals","Small disease model excels on triplet evaluation"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing assumption is that the synthetic training data generated from ICD-10-CM names does not overlap with the three evaluation datasets; the paper reports no deduplication or overlap analysis, so if GPT-4o-mini reproduced or paraphrased any test disease–symptom associations, the benchmark accuracy would reflect memorization rather than generalization.","fun_headline_variants_meta":{"raw":{"variants":["DisEmbed beats larger medical models on disease tasks","33M-parameter disease model outperforms medical giants","Synthetic data helps small embedding win disease benchmarks","Disease-focused embedding tops PubMedBERT and rivals","Small disease model excels on triplet evaluation"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00024,"raw_usage":{"total_tokens":1498,"prompt_tokens":903,"completion_tokens":595,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":519,"completion_tokens_details":{"reasoning_tokens":525}},"tokens_in":519,"tokens_out":595,"duration_ms":6668,"temperature":1.0,"reasoning_tokens":525,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T14:39:57.429794+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compute lexical and semantic overlap between the synthetic training corpus and each of the three evaluation datasets; if any test triplet's disease–symptom pairs appear verbatim or near-verbatim in training, the reported advantage must be re-measured on deduplicated data. A stronger test is to retrain DisEmbed-v1 on a version of the corpus that excludes all diseases appearing in the three benchmarks, then check whether the accuracy gap over the baseline models persists.","supporting_citations":[{"cited_title":"Medical diagnosis dialogue workshop @ iclr 2021","cited_arxiv_id":null,"evidence_quote":"Medical Diagnosis Dialogue dataset, the second benchmark in Table 1."},{"cited_title":"Cod-patientsymdisease","cited_arxiv_id":null,"evidence_quote":"CoD-PatientSymDisease, the third benchmark in Table 1."},{"cited_title":"Disease database","cited_arxiv_id":null,"evidence_quote":"Disease Database, the first of the three disease–symptom benchmarks in Table 1."},{"cited_title":"Generalist embedding models are better at short-context clinical semantic search than specialized embedding models, 2024","cited_arxiv_id":null,"evidence_quote":"The prior study whose ICD-10-based evaluation the paper argues is inadequate for measuring disease semantics."},{"cited_title":"International classification of diseases, tenth revision, clinical modification (icd-10-cm)","cited_arxiv_id":null,"evidence_quote":"Supplies the 70,000+ ICD-10-CM disease names used to generate the synthetic training corpus."},{"cited_title":"Neuml/pubmedbert-base-embeddings","cited_arxiv_id":null,"evidence_quote":"PubMedBERT, a larger medical embedding baseline that DisEmbed-v1 claims to outperform in Table 1."},{"cited_title":"Medembed: Pre-trained medical embeddings","cited_arxiv_id":null,"evidence_quote":"MedEmbed, the medical embedding model family used as a baseline in Table 1."},{"cited_title":"Efficient natural language response suggestion for smart reply, 2017","cited_arxiv_id":null,"evidence_quote":"Defines the triplet evaluation condition used for all benchmark accuracy scores."}],"review_version":1}