REVIEW 3 major objections 4 minor 26 references
Two-stage Federated Phenotyping and Patient Representation Learning
T0 review · 3 major / 4 minor · reviewed 2026-08-14 · deepseek-v4-flash
Pith's one-line read Federated clinical phenotyping can match centralized training (F1 0.724) without sharing patient notes.
desk verdict First federated clinical NLP proof-of-concept that is plausible but under-evidenced due to IID site simulation and missing statistics. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is federated averaging, applied twice. At stage 1, a deep averaging network (a network that averages token embeddings and passes them through a dense layer) takes each patient's clinical notes as standardized medical concept identifiers and is trained to predict ICD and CPT codes; local weights from each simulated site are combined as $W_{\mathrm{ag}} = \sum_{k} \frac{n_k}{N} W_k$, with $n_k$ the number of patients at site $k$. The frozen dense layer then maps any new patient's notes to a fixed-length vector. At stage 2, a linear-kernel support vector machine is trained on those vectors to classify each of the sixteen obesity-related phenotypes, again through local training plus sample-size-weighted averaging. The central identity the argument rests on is this weighted averaging update: the claim is that it preserves almost all of the accuracy of centralized training while letting each site keep its raw data.
What would settle it
Run the same two-stage pipeline with uneven, non-identical data across sites (for instance, split the i2b2 discharge summaries by institution or note style, or split the representation-learning corpus by admission year) and compare the federated F1 with centralized and single-site F1. If the federated F1 falls toward the single-site level, the paper's claim of parity with centralized training is refuted.
Extended reading notes
Core claim
The central discovery is that federated averaging, applied at both stages, preserves the accuracy of centralized training while keeping data in place. In the authors' experiments, a deep averaging network pre-trained to predict ICD and CPT codes from standardized medical concept identifiers, with weights averaged across ten simulated sites, produced patient representations that supported a federated linear-kernel support vector machine phenotype classifier across three simulated sites. The fully federated configuration achieved F1 0.724, which the paper states was not statistically different from the centralized two-stage baseline (experiment 4, F1 0.726 in Table 2), and exceeded single-source phenotyping (F1 0.542). The paper also reports that federated phenotyping without representation learning (F1 0.632) matched centralized phenotyping without representation learning (F1 0.634), showing the federated mechanism works at both stages independently.
Load-bearing premise
The load-bearing premise is that the simulated silos, created by randomly splitting patients, behave like real hospitals; if actual hospitals differ systematically in population, note style, or coding practice, the reported parity with centralized training may not survive.
Editorial extensions
If this is right
- Multi-site phenotyping becomes feasible without a central data repository: each site trains locally and only model weights are exchanged.
- Federated representation learning on large general corpora can transfer to smaller task-specific annotated sets, raising phenotype accuracy from 0.542 (single-site) to 0.724 (fully federated).
- Because stage 1 notes do not need target-phenotype labels, hospitals can contribute routine notes to representation learning without extra annotation effort.
- The same two-stage recipe applies to raw text or other feature representations, as the authors state, so it is not tied to standardized concept identifiers.
Reading between the lines
- A natural next step is to test the pipeline on naturally partitioned data, such as notes from different hospital systems with different demographics and documentation styles; the random-split simulation likely understates real-world heterogeneity.
- If site distributions differ enough, sample-size-weighted averaging may need per-site regularization or personalization, a possibility the paper does not explore.
- Combining this scheme with cryptographic aggregation or differential privacy could strengthen the privacy story, but the paper does not measure the resulting communication or accuracy costs.
- Per-disease results vary widely (for example, gout F1 0.938 versus GERD F1 0.558), so the averaged gain may not hold uniformly; identifying which phenotypes benefit most would sharpen the claim.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes a two-stage federated learning method for clinical NLP. Stage 1 pre-trains a deep averaging network on MIMIC-III clinical notes, using UMLS CUIs as input and ICD/CPT codes as labels, to obtain patient representations. Stage 2 trains a linear SVM phenotyping classifier on i2b2 obesity and comorbidity annotations, using the stage-1 representations as features. Both stages use federated averaging over simulated sites. The authors report seven experiments comparing centralized, federated, and single-site training. The headline result is that the fully federated two-stage pipeline (Experiment 7) achieves F1 0.724, which the text claims is not statistically different from centralized representation and centralized phenotyping (Experiment 4, reported as F1 0.726 in Table 2 and 0.714 in the text) and is better than single-site phenotyping (Experiment 3, F1 0.542). The paper also reports per-disease results for 16 obesity comorbidities.
Significance. If the headline result held with proper statistical support, the paper would be a useful early demonstration that clinical text can be used for representation learning and phenotyping without centralizing the data. The work has several strengths: the evaluation is an empirical comparison against held-out labels; the code is released on GitHub; the federated averaging is implemented with sample-size weighting; and the two-stage design allows the authors to decompose where federation preserves or loses performance. The main limitations are that the simulated sites are IID random splits, the key equivalence claim is supported by no statistical test, and the aggregate F1 averages over heterogeneous per-disease label schemes. These limitations directly affect the central claim, so the paper needs substantial revision before the conclusions can be accepted.
major comments (3)
- [4.1 / Table 2] The central equivalence claim — that Experiment 7 (F1 0.724) is "not statistically different" from Experiment 4 — is unsupported. No standard deviation, confidence interval, paired test statistic, or p-value is reported; "over multiple rounds of experiment using different data shuffling and initialization" is not a statistical test. The comparison is also numerically inconsistent: Table 2 reports Experiment 4 F1 as 0.726, while the text reports it as 0.714 and later reports 0.718/0.711/0.714 for centralized training at both stages. Because this equivalence is the paper's headline finding, the revision must report per-seed or per-disease variance with a named significance test and reconcile the conflicting numbers.
- [Section 3 / Algorithm 1] The simulated silos are created by randomly splitting patients (MIMIC-III into 10 sites and i2b2 into 3 sites), which produces near-IID partitions in which each site's local distribution approximates the global distribution. The motivating scenario in the abstract and introduction, however, is heterogeneity and uniqueness of medical documents across providers, and federated averaging is known to degrade under non-IID data. Experiment 7 therefore does not actually test the intended deployment scenario. The revision should add non-IID partitions (for example, label skew, quantity skew, or site-specific vocabulary/style shifts) or explicitly restrict the claim to IID data distributions.
- [4.1 / Table 3 / final paragraph] The per-disease results in Table 3 appear to mix 3-class and 2-class classification tasks. The text states that for diseases with few "questionable" cases, those cases were excluded and binary presence/absence classification was performed, while for other diseases the 3-class scheme was presumably retained. The macro-averaged F1 of 0.724 is therefore an average over non-identical classification tasks. The paper should state per disease which label scheme was used, and should either compare aggregate scores only over diseases with the same scheme or report separate averages for binary and 3-class subsets.
minor comments (4)
- [Section 2.6] The description of the phenotyping SVM is under-specified: a linear SVM with no regularization and trained with stochastic gradient descent needs a stated loss, learning rate, number of epochs, and any penalty or stopping criterion to be reproducible. The released code helps, but the paper should report these settings.
- [Section 2.7] Equation (1) is the standard FedAvg update, but the objective function displayed below it is written imprecisely: f(x_jp) and the indexing of p over outputs should be clarified, and the relationship between the ANN objective in stage 1 and the SVM objective in stage 2 should be stated explicitly.
- [Throughout] There are numerous typos and wording errors, including "curently" in the introduction, "instituitions" in the introduction, "clinicans" in Section 2.3, "comorbities" in the abstract, "comobidity" in Table 3, and "data form that site" in Section 2.7. The manuscript would benefit from a careful proofreading pass.
- [4.1] The sentence "In comparison, when only data from a single simulated silo was used, the average F1 score 0.634" is confusing: 0.634 is the F1 of Experiment 1 (centralized bag-of-CUIs), not a single-site result. The comparison between Experiment 3 and Experiment 1 should be stated more clearly.
Circularity Check
No significant circularity; central claims rest on held-out empirical comparisons rather than on fitted values or self-citation chains.
full rationale
The paper's central claims are empirical: Section 4 and Table 2 compare F1 scores across seven experiments against the i2b2 held-out labels. Federated averaging (Eq. 1) is a standard aggregation rule and is not used to define the evaluation metric or the phenotyping outcome. Self-citations, especially Dligach and Miller (2018), supply the patient-representation architecture, but the paper independently evaluates the benefit of learned representations in experiments 4, 6, and 7 rather than relying solely on the cited result. The random-split simulation of sites is a concern about external validity for non-IID clinical data, not a circularity: the reported equivalence between federated and centralized training is still an empirical observation, albeit one that may not generalize. The unsupported statement that experiment 7 is 'not statistically different' from experiment 4 is an evidentiary weakness, not a case of a prediction being equal to its input by construction. No fitted parameter is renamed as a prediction, and no uniqueness or ansatz is imported through self-citation to force the outcome. Therefore no circular step is present.
Assumptions & free parameters
assumptions (5)
- domain assumption Randomly splitting patients into silos approximates real-world distributed hospital data, with each site's local distribution representative of the global distribution.
- domain assumption UMLS CUIs extracted by cTAKES contain sufficient information for patient representation and phenotyping.
- domain assumption Predicting ICD and CPT codes from notes is a useful pretraining objective for patient representations.
- domain assumption Federated averaging with sample-size weights converges close enough to centralized training for finite communication rounds.
- domain assumption The i2b2 intuitive annotations of presence, absence, and questionable are reliable ground truth for the phenotyping task.
Cite this review
Pith. "Pith review of Two-stage Federated Phenotyping and Patient Representation Learning." pith.science (2026). https://pith.science/paper/TJWXRKZJ
@misc{pith2026190805596,
author = {Pith},
title = {Pith review of: Two-stage Federated Phenotyping and Patient Representation Learning},
year = {2026},
howpublished = {\url{https://pith.science/paper/TJWXRKZJ}},
note = {Machine review of arXiv:1908.05596}
}
read the original abstract
A large percentage of medical information is in unstructured text format in electronic medical record systems. Manual extraction of information from clinical notes is extremely time consuming. Natural language processing has been widely used in recent years for automatic information extraction from medical texts. However, algorithms trained on data from a single healthcare provider are not generalizable and error-prone due to the heterogeneity and uniqueness of medical documents. We develop a two-stage federated natural language processing method that enables utilization of clinical notes from different hospitals or clinics without moving the data, and demonstrate its performance using obesity and comorbities phenotyping as medical task. This approach not only improves the quality of a specific clinical task but also facilitates knowledge progression in the whole healthcare system, which is an essential part of learning health system. To the best of our knowledge, this is the first application of federated machine learning in clinical NLP.
Figures
Reference graph
Works this paper leans on
-
[1]
Amy P Abernethy, Lynn M Etheredge, Patricia A Ganz, Paul Wallace, Robert R German, Chalapathy Neti, Peter B Bach, and Sharon B Murphy. 2010. Rapid-learning system for cancer care. Journal of Clinical Oncology, 28(27):4268
work page 2010
-
[2]
David Blumenthal and Marilyn Tavenner. 2010. The “meaningful use” regulation for electronic health records. New England Journal of Medicine, 363(6):501--504
work page 2010
-
[3]
Keith Bonawitz, Hubert Eichner, Wolfgang Grieskamp, Dzmitry Huba, Alex Ingerman, Vladimir Ivanov, Chloe Kiddon, Jakub Konecny, Stefano Mazzocchi, H Brendan McMahan, et al. 2019. Towards federated learning at scale: System design. arXiv preprint arXiv:1902.01046
arXiv 2019
-
[4]
Mike Conway, Richard L Berg, David Carrell, Joshua C Denny, Abel N Kho, Iftikhar J Kullo, James G Linneman, Jennifer A Pacheco, Peggy Peissig, Luke Rasmussen, et al. 2011. Analyzing the heterogeneity and complexity of electronic health record oriented phenotyping algorithms. In AMIA annual symposium proceedings, volume 2011, page 274. American Medical Inf...
work page 2011
-
[5]
Dina Demner-Fushman, Wendy W Chapman, and Clement J McDonald. 2009. What can natural language processing do for clinical decision support? Journal of biomedical informatics, 42(5):760--772
2009
-
[6]
Dmitriy Dligach and Timothy Miller. 2018. Learning patient representations from text. arXiv preprint arXiv:1805.02096
work page Pith review arXiv 2018
-
[7]
Elizabeth Ford, John A Carroll, Helen E Smith, Donia Scott, and Jackie A Cassell. 2016. Extracting information from the text of electronic medical records to improve case detection: a systematic review. Journal of the American Medical Informatics Association, 23(5):1007--1015
work page 2016
-
[8]
Carol Friedman, Lyudmila Shagina, Yves Lussier, and George Hripcsak. 2004. Automated encoding of clinical documents based on natural language processing. Journal of the American Medical Informatics Association, 11(5):392--402
work page 2004
Show all 26 references
-
[9]
Charles Friedman, Joshua Rubin, Jeffrey Brown, Melinda Buntin, Milton Corn, Lynn Etheredge, Carl Gunter, Mark Musen, Richard Platt, William Stead, et al. 2014. Toward a science of learning systems: a research agenda for the high-functioning learning health system. Journal of t...
2014
-
[10]
Charles P Friedman, Adam K Wong, and David Blumenthal. 2010. Achieving a nationwide learning health system. Science translational medicine, 2(57):57cm29--57cm29
2010
-
[11]
Li Huang and Dianbo Liu. 2019. Patient clustering improves efficiency of federated machine learning to predict mortality and hospital stay time using distributed electronic medical records. arXiv preprint arXiv:1903.09296
2019 arXiv
-
[12]
Li Huang, Yifeng Yin, Zeng Fu, Shifa Zhang, Hao Deng, and Dianbo Liu. 2018. Loadaboost: Loss-based adaboost federated machine learning on medical data. arXiv preprint arXiv:1811.12629
2018 arXiv
-
[13]
Vasudevan Jagannathan, Charles J Mullett, James G Arbogast, Kevin A Halbritter, Deepthi Yellapragada, Sushmitha Regulapati, and Pavani Bandaru. 2009. Assessment of commercial nlp engines for medication information extraction from dictated clinical notes. International journal ...
2009
-
[14]
Alistair EW Johnson, Tom J Pollard, Lu Shen, H Lehman Li-wei, Mengling Feng, Mohammad Ghassemi, Benjamin Moody, Peter Szolovits, Leo Anthony Celi, and Roger G Mark. 2016. Mimic-iii, a freely accessible critical care database. Scientific data, 3:160035
2016
-
[15]
Jakub Kone c n \`y , H Brendan McMahan, Felix X Yu, Peter Richt \'a rik, Ananda Theertha Suresh, and Dave Bacon. 2016. Federated learning: Strategies for improving communication efficiency. arXiv preprint arXiv:1610.05492
2016 arXiv
-
[16]
Kory Kreimeyer, Matthew Foster, Abhishek Pandey, Nina Arya, Gwendolyn Halford, Sandra F Jones, Richard Forshee, Mark Walderhaug, and Taxiarchis Botsis. 2017. Natural language processing systems for capturing and standardizing unstructured clinical information: a systematic rev...
2017
-
[17]
Dianbo Liu, Timothy Miller, Raheel Sayeed, and Kenneth Mandl. 2018 a . Fadl: Federated-autonomous deep learning for distributed electronic health record. arXiv preprint arXiv:1811.11400
2018 arXiv
-
[18]
Dianbo Liu, Nestor Sepulveda, and Ming Zheng. 2018 b . Artificial neural networks condensation: A strategy to facilitate adaption of machine learning in medical settings by reducing computational burden. arXiv preprint arXiv:1812.09659
2018 arXiv
-
[19]
Christopher D Manning, Christopher D Manning, and Hinrich Sch \"u tze. 1999. Foundations of statistical natural language processing. MIT press
1999
-
[20]
H Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, et al. 2016. Communication-efficient learning of deep networks from decentralized data. arXiv preprint arXiv:1602.05629
2016 arXiv
-
[21]
Riccardo Miotto, Li Li, Brian A Kidd, and Joel T Dudley. 2016. Deep patient: an unsupervised representation to predict the future of patients from the electronic health records. Scientific reports, 6:26094
2016
-
[22]
Harvey J Murff, Fern FitzHenry, Michael E Matheny, Nancy Gentry, Kristen L Kotter, Kimberly Crimin, Robert S Dittus, Amy K Rosen, Peter L Elkin, Steven H Brown, et al. 2011. Automated identification of postoperative complications within an electronic medical record using natur...
2011
-
[23]
Jean R Slutsky. 2007. Moving closer to a rapid-learning health care system. Health affairs, 26(2):w122--w124
2007
-
[24]
\"O zlem Uzuner. 2009. Recognizing obesity and comorbidities in sparse data. Journal of the American Medical Informatics Association, 16(4):561--570
2009
-
[25]
URL: " 'urlintro :=
ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before...
-
[26]
write newline
" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...
Reviewed August 14, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.