REVIEW 4 major objections 5 minor 1 cited by
Structured Semantics from Unstructured Notes: Language Model Approaches to EHR-Based Decision Support
T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read The paper argues that language models reading clinical notes can extract representations that beat structured EHR features and transfer across institutions, though its supporting tables are hypothetical.
desk verdict An unfinished draft whose only results are explicitly hypothetical, despite an abstract claiming 'we demonstrate'; not ready for peer review. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is the contextualized text embedding produced by a pretrained biomedical language model, pooled into a single vector per clinical note. Transformer self-attention computes each token's representation as a weighted sum over the note, so words are understood in context rather than as isolated features. On top of the plain text encoder, the paper adds a second mechanism—medical-code integration—where a model is trained to keep the embedding of a code close to the embedding of its textual description (for example, concatenating 'code: Ck description: D(Ck)' into the input or using a contrastive loss), which grounds the free-text space in structured ontology semantics. Parameter-efficient fine-tuning via low-rank adaptation (LoRA) supplies the practical third mechanism, making it feasible to adapt these large encoders in clinical settings. Together the text encoder, the code-description alignment, and cheap fine-tuning carry the argument that hybrid text-plus-code representations are accurate and institution-invariant.
What would settle it
Run the paper's claimed comparison on a real multi-institution EHR corpus: train Clinical ModernBERT on text plus ICD codes from one hospital and XGBoost on structured features from the same hospital, then test both on a held-out hospital's data. The central claim fails if the text-based model does not beat the structured-feature baseline on the external site by a clinically meaningful margin, or if the text embedding shows the same institutional shift as the code distributions.
Extended reading notes
Core claim
On its own terms, the paper claims that semantically informed text features are the missing ingredient in EHR machine learning: models that consume clinical notes through pretrained language models outperform models limited to structured EHR fields (Table 2), generalize to unseen institutions with a smaller performance drop (Table 3), and gain further from training that pairs medical codes with their natural-language descriptions (Table 4). It identifies Clinical ModernBERT as the strongest configuration, with a hypothetical AUC-ROC of 0.91 for diagnostic classification versus 0.85 for XGBoost on structured data, and attributes the gap to the model's joint encoding of long clinical context and coded concepts. The paper's central claim is thus that unstructured notes are not a nuisance modality to be discarded but the semantically richest view of the patient record, and that aligning text with ontologies such as ICD codes makes the representation both more accurate and more portable.
Load-bearing premise
The load-bearing premise is that the hypothetical numbers in the results tables—such as Clinical ModernBERT at 0.91 AUC against XGBoost's 0.85—are representative of what real experiments on real EHR data would find.
Editorial extensions
If this is right
- If text features carry the claimed signal, clinical NLP should become a standard column of EHR risk prediction rather than an optional extra.
- Cross-institution deployment could shift from per-site retraining to shared text-based encoders that harmonize documentation differences.
- Pretraining recipes for clinical models should routinely pair medical codes with their textual descriptions, since the paper's tables tie code-description supervision to higher accuracy.
- Evaluation of clinical models should adopt the paper's proposed norms: bootstrapped confidence intervals, subgroup-stratified reporting, and institution-disjoint splits.
Reading between the lines
- Because the reported gains are hypothetical, a fair reading is that the paper stakes out a testable priority: text-derived features will beat structured features at external sites. A direct reimplementation, which the paper does not provide, is the natural next step.
- If note text truly is the harmonizing signal, the benefit should be weakest for hospitals with terse or fully template-driven notes, so the claim implies a boundary condition the paper does not explore.
- Code-description alignment might generalize beyond ICD to ontologies the model never saw, since the representation is learned from language rather than from a fixed code table; this extension is not tested in the paper.
- The hypothetical comparison suggests a concrete threshold a real study must beat: Clinical ModernBERT at 0.91 AUC versus XGBoost at 0.85, with the external-transfer gap similarly quantified.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper argues that language models applied to free-text clinical notes can improve predictive performance and cross-institution generalization in EHR-based decision support, and that augmenting them with medical codes (ICD, CPT) through approaches such as Clinical ModernBERT yields further gains. It surveys relevant model families, describes standard methods for feature extraction, code integration, parameter-efficient fine-tuning with LoRA, and evaluation practices, and then presents numerical results in Section 4 that are explicitly labeled "hypothetical results." The discussion covers limitations and future directions, including interpretability, generalization, fairness, and synthetic data.
Significance. If supported by real experiments, the claim that text-based LM features outperform structured EHR features and generalize across institutions would be practically important for clinical decision support. The paper competently summarizes relevant literature and standard modeling techniques, and it correctly recommends rigorous evaluation practices such as bootstrap confidence intervals, subgroup analysis, and cross-institutional testing. However, the manuscript contains no real empirical evaluation: all quantitative results are explicitly hypothetical, and key premises about the advantages of Clinical ModernBERT and code-description integration are drawn from the authors' own prior work. Consequently, the central empirical claims are not established in this manuscript; as a tutorial or position paper it would need to be reframed, and as an empirical study it is currently not significant.
major comments (4)
- [Section 4, Tables 2-4; Abstract] The abstract states "We demonstrate that language models can extract meaningful representations from unstructured notes that not only improve predictive performance, but also generalize more effectively across institutional boundaries," but Section 4 opens with "we present hypothetical results" and Tables 2-4 are captioned "(Hypothetical Results)." The numbers that are supposed to support the paper's central claims (e.g., Clinical ModernBERT AUC 0.91 vs. XGBoost 0.85 in Table 2; 0.83 vs. 0.72 cross-institution in Table 3) are therefore invented illustrations, not reported measurements. No dataset splits, model checkpoints, or code are provided. The word "demonstrate" is thus unsupported; either real experiments must be reported or the claim must be explicitly downgraded to a proposal.
- [Appendix A.4, Table 5] Table 5 is presented as an "Ablation Study" with the sentence "Results are averaged over 3 seeds" and no "hypothetical" label, unlike Tables 2-4. This unlabeled table presents the same kind of fictional numbers as though they were real experimental outcomes, blurring the distinction between illustration and evidence. The table must be explicitly labeled as hypothetical or replaced with actual runs, including the variance across seeds.
- [Section 5.1] The limitations paragraph concedes that generalization "can degrade significantly when applied to different health systems, even when leveraging textual features" and that clinical narratives "are shaped by institutional culture, billing practices, and individual provider styles." These statements directly contradict the abstract's claim that language models generalize "more effectively across institutional boundaries" and the cross-institution gains shown in Table 3. The manuscript must either provide evidence that the text-based advantage survives realistic distribution shift or soften the central claim accordingly.
- [Sections 2 and 3.3] The key premises about the superiority of text-based representations and the benefit of code-description integration are drawn from the authors' own prior works (Lee et al., 2024a; Lee et al., 2025), and the paper does not independently test them. For a survey this would be acceptable, but then the abstract should not claim to "demonstrate" those results; the distinction between summarizing prior work and presenting new evidence must be made explicit in the text.
minor comments (5)
- [Section 3.1] There are placeholder citations such as "[cite: 22]" and "[cite: 30]" that should be replaced with proper references.
- [References] The reference list has incomplete entries: "Vaswani, A. Attention is all you need" and "Devlin, J. BERT: Pre-training..." omit co-authors and should be formatted as Vaswani et al. and Devlin et al.
- [Section 2] In the sentence about commercial-scale LLMs, "OpenAI's 01 system" should read "OpenAI's O1 system."
- [Section 3.3] The contrastive loss function Lcode is mentioned but never explicitly defined; provide the equation or remove the reference to a specific loss.
- [Appendix A.4, Table 5] If Table 5 were real data, the paper's own evaluation framework in Section 3.5 would require confidence intervals or standard errors rather than only point estimates averaged over three seeds.
Circularity Check
The 'demonstration' of text-based superiority and cross-institution generalization rests entirely on numbers the paper itself labels '(Hypothetical Results)'; the evidence is authored to match the thesis, and Appendix A.4 re-labels the same invented values as a real 'ablation study.'
-
fitted input called prediction
[Section 4 (Results and captions of Tables 2-4); Abstract; Section 5]
"We demonstrate that language models can extract meaningful representations from unstructured notes that not only improve predictive performance, but also generalize more effectively across institutional boundaries. ... In this section, we present hypothetical results from applying language models to EHR data for various clinical prediction tasks. ... Our hypothetical experiments demonstrate that models trained on free-text clinical notes, especially those incorporating medical codes, consistently outperform or complement traditional approaches that rely solely on structured EHR data."
The abstract's 'demonstrate' is supported only by the numbers in Tables 2-4, which the paper itself captions '(Hypothetical Results)' and describes as 'hypothetical experiments.' No dataset, data split, model checkpoint, or measured pipeline is supplied; the AUC/F1 values (e.g., Clinical ModernBERT 0.91 vs. XGBoost 0.85) were chosen by the authors to illustrate the thesis. The conclusion that text features outperform and generalize more effectively is therefore equivalent, by construction, to the invented table entries: the evidence was authored to match the claim, so the claimed demonstration reduces to its own input.
-
other
[Appendix A.4, Table 5]
"We performed an ablation study to quantify the effect of including medical code descriptions during pretraining. All experiments were conducted on the same downstream disease prediction task using Clinical ModernBERT. Results are averaged over 3 seeds. ... Table 5. Ablation Study: Impact of Code Description Integration ... Clinical ModernBERT (with codes + descriptions) 0.91 0.86 0.89"
The appendix presents the same style of table and the same Clinical ModernBERT row (AUC 0.91) as a real 'ablation study' with results 'averaged over 3 seeds,' while omitting the '(Hypothetical Results)' label used in Section 4. This converts the paper's own explicitly hypothetical values into apparent experimental evidence after seed averaging, making the circular support look like independent validation of code-description integration. The unlabeled table blurs the real/hypothetical boundary and repackages the same authored inputs as measurement.
full rationale
The paper's central empirical claim—that language-model text features improve predictive performance and cross-institution generalization—has no measured support inside the manuscript. Section 4 states 'we present hypothetical results,' and Tables 2-4 carry '(Hypothetical Results)' in their captions, so the quantitative content (text AUC 0.89-0.91 vs. XGBoost 0.85; cross-institution 0.83 vs. 0.72) is not a report of experiments but a set of numbers chosen to illustrate the thesis. The conclusion is thus forced by construction: the 'results' are the input, not an output of a derivation chain. Appendix A.4 deepens the problem by presenting an unlabeled 'ablation study ... averaged over 3 seeds' with the same invented values, which invites the reader to treat hypothetical inputs as real experimental evidence. No independent fit, external benchmark, or code/data release breaks this circularity. Regarding the cited Clinical ModernBERT (Lee et al., 2025) and DK-BEHR (An et al., 2025) works: these are external premises used to motivate the architecture, but they are not clearly self-citations of the current author list, so they are not separately scored as load-bearing self-citation. The circularity score is high because the main 'demonstration' reduces, by the paper's own labeling, to the hypothetical numbers the authors supplied.
Assumptions & free parameters
free parameters (1)
- Hypothetical performance numbers in Tables 2-4 and Table 5 =
e.g., Clinical ModernBERT AUC 0.91, F1 0.86, Accuracy 0.89; XGBoost AUC 0.85
assumptions (3)
- domain assumption Pre-trained clinical language models such as ClinicalBERT and Clinical ModernBERT produce representations that capture clinically meaningful semantics from text.
- domain assumption Explicitly integrating medical codes and their descriptions during pretraining improves code understanding and downstream performance.
- ad hoc to paper The hypothetical performance numbers in Tables 2-4 are representative of real-world experiments.
Cite this review
Pith. "Pith review of Structured Semantics from Unstructured Notes: Language Model Approaches to EHR-Based Decision Support." pith.science (2026). https://pith.science/paper/WFR4IYYO
@misc{pith2026250606340,
author = {Pith},
title = {Pith review of: Structured Semantics from Unstructured Notes: Language Model Approaches to EHR-Based Decision Support},
year = {2026},
howpublished = {\url{https://pith.science/paper/WFR4IYYO}},
note = {Machine review of arXiv:2506.06340}
}
read the original abstract
The advent of large language models (LLMs) has opened new avenues for analyzing complex, unstructured data, particularly within the medical domain. Electronic Health Records (EHRs) contain a wealth of information in various formats, including free text clinical notes, structured lab results, and diagnostic codes. This paper explores the application of advanced language models to leverage these diverse data sources for improved clinical decision support. We will discuss how text-based features, often overlooked in traditional high dimensional EHR analysis, can provide semantically rich representations and aid in harmonizing data across different institutions. Furthermore, we delve into the challenges and opportunities of incorporating medical codes and ensuring the generalizability and fairness of AI models in healthcare.
Forward citations
Cited by 1 Pith paper
-
Expanders Meet Reed-Muller: Easy Instances of Noisy k-XOR
Explicit near-optimal expanders exist for which noisy k-XOR is polynomial-time solvable, falsifying conjectures that expansion implies hardness.
Reference graph
Works this paper leans on
-
[1]
L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al
Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al. Gpt-4 technical report.arXiv preprint arXiv:2303.08774,
-
[5]
7 Submission and Formatting Instructions for ICML 2025 Evans, R. S. Electronic health records: then, now, and in the future.Yearbook of medical informatics, 25(S 01): S48–S61,
work page 2025
-
[10]
URL http://arxiv.org/ abs/1812.02275. Lee, S. A. and Lindsey, T. Do large language models under- stand medical codes?arXiv preprint arXiv:2403.10822,
-
[11]
A., Jain, S., Chen, A., Ono, K., Fang, J., Rudas, A., and Chiang, J
Lee, S. A., Jain, S., Chen, A., Ono, K., Fang, J., Rudas, A., and Chiang, J. N. Emergency department deci- sion support using clinical pseudo-notes.arXiv preprint arXiv:2402.00160, 2024a. Lee, S. A., Lee, J., and Chiang, J. N. Feet: A framework for evaluating embedding techniques.arXiv preprint arXiv:2411.01322, 2024b. Lee, S. A., Wu, A., and Chiang, J. N...
-
[12]
Lin, Y ., Yu, Z. B., and Lee, S. A case study exploring the current landscape of synthetic medical record generation with commercial llms.arXiv preprint arXiv:2504.14657,
-
[13]
Team, G., Anil, R., Borgeaud, S., Alayrac, J.-B., Yu, J., Sori- cut, R., Schalkwyk, J., Dai, A
URL https://arxiv.org/abs/2305.09617. Team, G., Anil, R., Borgeaud, S., Alayrac, J.-B., Yu, J., Sori- cut, R., Schalkwyk, J., Dai, A. M., Hauth, A., Millican, K., et al. Gemini: a family of highly capable multimodal models.arXiv preprint arXiv:2312.11805,
-
[15]
Large language models for disease diagnosis: A scoping review.arXiv preprint arXiv:2409.00097,
Zhou, S., Xu, Z., Zhang, M., Xu, C., Guo, Y ., Zhan, Z., Ding, S., Wang, J., Xu, K., Fang, Y ., et al. Large language models for disease diagnosis: A scoping review.arXiv preprint arXiv:2409.00097,
-
[16]
8 Submission and Formatting Instructions for ICML 2025 A. Appendix A.1. Implementation Details Our implementation uses PyTorch 2.1. We fine-tuned all models using the AdamW optimizer with linear warm-up followed by cosine decay. Unless otherwise stated, we set the learning rate to 2e-5 for language models and 1e-3 for classification heads. Batch size was ...
work page 2025
Show all 16 references
-
[2016]
Johnson, A
doi: https://doi.org/10.1038/sdata.2016.35. Johnson, A. E. W., Pollard, T. J., and Naumann, T. Gen- eralizability of predictive models for intensive care unit patients.Machine Learning for Health (ML4H) Workshop at NeurIPS 2018,
2016 doi
-
[2017]
J., Shen, Y ., Wallis, P., Allen-Zhu, Z., Li, Y ., Wang, S., Wang, L., and Chen, W
Hu, E. J., Shen, Y ., Wallis, P., Allen-Zhu, Z., Li, Y ., Wang, S., Wang, L., and Chen, W. Lora: Low-rank adaptation of large language models.arXiv preprint arXiv:2106.09685,
-
[2018]
Bert: Pre-training of deep bidirectional trans- formers for language understanding.arXiv preprint arXiv:1810.04805,
Devlin, J. Bert: Pre-training of deep bidirectional trans- formers for language understanding.arXiv preprint arXiv:1810.04805,
-
[2021]
Openai o1 system card.arXiv preprint arXiv:2412.16720,
Jaech, A., Kalai, A., Lerer, A., Richardson, A., El-Kishky, A., Low, A., Helyar, A., Madry, A., Beutel, A., Car- ney, A., et al. Openai o1 system card.arXiv preprint arXiv:2412.16720,
-
[2022]
Attention is all you need.arXiv preprint arXiv:1706.03762,
Vaswani, A. Attention is all you need.arXiv preprint arXiv:1706.03762,
-
[2023]
R., Boag, W., Weng, W.-H., Jin, D., Naumann, T., and McDermott, M
Alsentzer, E., Murphy, J. R., Boag, W., Weng, W.-H., Jin, D., Naumann, T., and McDermott, M. Publicly available clin- ical bert embeddings.arXiv preprint arXiv:1904.03323,
1904 arXiv
-
[2024]
A., and Mark, R
Johnson, A., Bulgarelli, L., Pollard, T., Horng, S., Celi, L. A., and Mark, R. Mimic-iv.Phy- sioNet. Available online at: https://physionet. org/content/mimiciv/1.0/(accessed August 23, 2021), pp. 49–55,
2021
-
[2025]
A., McDermott, M
Arnrich, B., Choi, E., Fries, J. A., McDermott, M. B., Oh, J., Pollard, T. J., Shah, N., Steinberg, E., Wornow, M., and van de Water, R. Medical event data standard (meds): Facilitating machine learning for health. Beltagy, I., Lo, K., and Cohan, A. Scibert: A pre- trained lan...
1903 arXiv
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.