{"id":"58c87f12-37ac-4221-b24d-0326ad4522c3","arxiv_id":"1908.05596","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Federated training of patient representations and phenotyping classifiers on clinical notes performs comparably to centralized training and better than training at one site.","lead":"Hospitals often cannot share medical records, so the authors tested a way to train AI models on notes spread across multiple hospitals without moving the notes. The method matched central training accuracy on an obesity complication detection task, suggesting a path toward privacy-preserving medical AI.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"IID site simulation does not test the motivating non-IID scenario; equivalence may not hold for heterogeneous hospitals.","rationale":"I read the paper in good faith. The two-stage approach is clearly described, code is available, and the experiments address a meaningful privacy-preserving NLP task. The reader's verdict is CONDITIONAL, and I agree with the weakest assumption: the simulated silos are IID, not reflecting real-world heterogeneity. My stress-test focuses on this because the paper's entire contribution is federated use of data from 'different hospitals or clinics,' and the experimental design does not exercise distribution shift. The reported equivalence is therefore not yet evidence for the central claim. I also note the internal inconsistency in the centralized F1 (0.726 in Table 2 vs 0.714 in text) and the unsupported 'not statistically different' statement, but these are secondary; the IID/non-IID issue is the most load-bearing. A concrete non-IID experiment would settle whether the concern lands. Until then, conditional acceptance with a request for non-IID evaluation is appropriate.","tokens_in":8397,"tokens_out":4932,"duration_ms":47041,"concrete_test":"Re-run Experiment 7 with non-IID site partitions: e.g., split MIMIC-III into 10 silos stratified by ICU service or admission year, and i2b2 into 3 silos by note source or by obesity prevalence; use the same architecture and training procedure. Report global and per-site F1 with 95% confidence intervals over multiple seeds for federated and centralized training. If the federated F1 falls more than 5 points below centralized (or the confidence intervals show a significant gap), the IID split is load-bearing and the equivalence claim does not generalize; if the gap remains within noise, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that two-stage federated NLP matches centralized training and beats single-site training, thereby enabling use of distributed clinical notes without data movement. Section 3 describes simulating sites by 'randomly split patients' (MIMIC-III into 10 sites, i2b2 into 3 sites). Random splits produce IID partitions: each site's local distribution approximates the global distribution, so federated averaging (Eq. 1) is expected to match centralized training regardless of algorithm robustness. The motivating scenario in the abstract and introduction is 'heterogeneity and uniqueness of medical documents' across providers; real hospitals differ in population, documentation style, and coding practice. No experiment uses non-IID partitions (e.g., site-specific vocabularies, label skew, quantity skew). Consequently, the reported equivalence of Experiment 7 (F1 0.724) and Experiment 4 (F1 0.726/0.714) may be an artifact of the simulation, not evidence that federated learning tolerates real-world heterogeneity. The paper's assertion that the two are 'not statistically different' is also unsupported by any variance or significance test. If federated averaging degrades under non-IID data—as is common in federated learning—the central claim fails for the intended deployment.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a two-stage federated learning method for clinical NLP. Stage 1 pre-trains a deep averaging network on MIMIC-III clinical notes, using UMLS CUIs as input and ICD/CPT codes as labels, to obtain patient representations. Stage 2 trains a linear SVM phenotyping classifier on i2b2 obesity and comorbidity annotations, using the stage-1 representations as features. Both stages use federated averaging over simulated sites. The authors report seven experiments comparing centralized, federated, and single-site training. The headline result is that the fully federated two-stage pipeline (Experiment 7) achieves F1 0.724, which the text claims is not statistically different from centralized representation and centralized phenotyping (Experiment 4, reported as F1 0.726 in Table 2 and 0.714 in the text) and is better than single-site phenotyping (Experiment 3, F1 0.542). The paper also reports per-disease results for 16 obesity comorbidities.","tokens_in":8568,"tokens_out":5925,"duration_ms":59463,"significance":"If the headline result held with proper statistical support, the paper would be a useful early demonstration that clinical text can be used for representation learning and phenotyping without centralizing the data. The work has several strengths: the evaluation is an empirical comparison against held-out labels; the code is released on GitHub; the federated averaging is implemented with sample-size weighting; and the two-stage design allows the authors to decompose where federation preserves or loses performance. The main limitations are that the simulated sites are IID random splits, the key equivalence claim is supported by no statistical test, and the aggregate F1 averages over heterogeneous per-disease label schemes. These limitations directly affect the central claim, so the paper needs substantial revision before the conclusions can be accepted.","major_comments":[{"comment":"The central equivalence claim — that Experiment 7 (F1 0.724) is \"not statistically different\" from Experiment 4 — is unsupported. No standard deviation, confidence interval, paired test statistic, or p-value is reported; \"over multiple rounds of experiment using different data shuffling and initialization\" is not a statistical test. The comparison is also numerically inconsistent: Table 2 reports Experiment 4 F1 as 0.726, while the text reports it as 0.714 and later reports 0.718/0.711/0.714 for centralized training at both stages. Because this equivalence is the paper's headline finding, the revision must report per-seed or per-disease variance with a named significance test and reconcile the conflicting numbers.","section":"4.1 / Table 2"},{"comment":"The simulated silos are created by randomly splitting patients (MIMIC-III into 10 sites and i2b2 into 3 sites), which produces near-IID partitions in which each site's local distribution approximates the global distribution. The motivating scenario in the abstract and introduction, however, is heterogeneity and uniqueness of medical documents across providers, and federated averaging is known to degrade under non-IID data. Experiment 7 therefore does not actually test the intended deployment scenario. The revision should add non-IID partitions (for example, label skew, quantity skew, or site-specific vocabulary/style shifts) or explicitly restrict the claim to IID data distributions.","section":"Section 3 / Algorithm 1"},{"comment":"The per-disease results in Table 3 appear to mix 3-class and 2-class classification tasks. The text states that for diseases with few \"questionable\" cases, those cases were excluded and binary presence/absence classification was performed, while for other diseases the 3-class scheme was presumably retained. The macro-averaged F1 of 0.724 is therefore an average over non-identical classification tasks. The paper should state per disease which label scheme was used, and should either compare aggregate scores only over diseases with the same scheme or report separate averages for binary and 3-class subsets.","section":"4.1 / Table 3 / final paragraph"}],"minor_comments":[{"comment":"The description of the phenotyping SVM is under-specified: a linear SVM with no regularization and trained with stochastic gradient descent needs a stated loss, learning rate, number of epochs, and any penalty or stopping criterion to be reproducible. The released code helps, but the paper should report these settings.","section":"Section 2.6"},{"comment":"Equation (1) is the standard FedAvg update, but the objective function displayed below it is written imprecisely: f(x_jp) and the indexing of p over outputs should be clarified, and the relationship between the ANN objective in stage 1 and the SVM objective in stage 2 should be stated explicitly.","section":"Section 2.7"},{"comment":"There are numerous typos and wording errors, including \"curently\" in the introduction, \"instituitions\" in the introduction, \"clinicans\" in Section 2.3, \"comorbities\" in the abstract, \"comobidity\" in Table 3, and \"data form that site\" in Section 2.7. The manuscript would benefit from a careful proofreading pass.","section":"Throughout"},{"comment":"The sentence \"In comparison, when only data from a single simulated silo was used, the average F1 score 0.634\" is confusing: 0.634 is the F1 of Experiment 1 (centralized bag-of-CUIs), not a single-site result. The comparison between Experiment 3 and Experiment 1 should be stated more clearly.","section":"4.1"}],"recommendation":"major_revision","confidential_remarks":"The paper is a short demonstration of a useful idea, and the empirical comparisons are honest in spirit. However, the headline claims currently rest on an unsupported statistical assertion and an IID simulation that does not match the motivating non-IID scenario. I would ask for a revision that reports repeated-seed statistics with a named test and adds at least one non-IID partition experiment. The novelty claim about being the first federated clinical NLP application should also be checked against the broader federated NLP literature, since the reference list is heavily dominated by self-citations."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague, here's my read on 1908.05596.\n\nThe new thing is the application of federated learning to clinical NLP end-to-end: a two-stage pipeline where both patient representation learning (on MIMIC-III) and phenotyping (on i2b2) are trained in a federated manner. As far as I can tell, that's a first in 2019. The experimental design is straightforward: seven conditions varying representation and phenotyping training centralized vs federated vs single-site. The headline result—federated two-stage (F1 0.724) is comparable to centralized (0.726) and clearly better than single-site (0.542)—is plausible and directionally correct.\n\nWhat the paper doesn't do is support its strongest claims. First, the 'not statistically different' statement appears without any variance, confidence interval, or test; we're given no data from the 'multiple rounds' they say they ran. Second, the simulated sites are IID random splits of patients. Real hospitals differ systematically in population, note style, and label distribution; under non-IID partitions federated averaging can degrade. So the equivalence result may be an artifact of the simulation. Third, there's a numeric inconsistency: Table 2 lists exp 4 F1 as 0.726, but the text reports 0.714 (and also gives different precision/recall). That needs reconciliation before I'd trust the tables. Fourth, the exclusion of 'questionable' cases for many diseases turns the task into binary classification, and the authors honestly say the results aren't comparable to the i2b2 challenge—fine, but the evaluation is then a different benchmark than the one the field knows.\n\nWho's this for? Researchers in privacy-preserving clinical NLP and learning health systems. It's a useful proof-of-concept and a clear benchmark reference, with code on GitHub. The flaws are fixable: run non-IID partitions (by hospital, by label), report variance with enough rounds, and clean up the numbers.\n\nRecommendation: send it to peer review. The idea deserves referee time, and the evaluation can be strengthened in revision. Not a desk reject; not a badge of honor either at this stage.","headline":"First federated clinical NLP proof-of-concept that is plausible but under-evidenced due to IID site simulation and missing statistics.","tokens_in":9081,"tokens_out":2397,"would_cite":true,"duration_ms":21753,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Federated clinical phenotyping can match centralized training (F1 0.724) without sharing patient notes.","keywords":["federated learning","clinical NLP","patient representation learning","phenotyping","obesity comorbidities","deep averaging network","MIMIC-III","i2b2"],"falsifier":"Run the same two-stage pipeline with uneven, non-identical data across sites (for instance, split the i2b2 discharge summaries by institution or note style, or split the representation-learning corpus by admission year) and compare the federated F1 with centralized and single-site F1. If the federated F1 falls toward the single-site level, the paper's claim of parity with centralized training is refuted.","tokens_in":8186,"feed_emoji":"🏥","tokens_out":9319,"duration_ms":84303,"temperature":0.7,"pith_summary":"The paper tries to show that clinical NLP can work across hospitals without moving patient notes. It trains patient representations in a first federated stage on a large critical-care corpus, then trains phenotype classifiers in a second federated stage on smaller annotated discharge summaries. The central result is that the fully federated two-stage pipeline reaches F1 0.724 on obesity-comorbidity phenotyping, which the paper reports is not statistically different from centralized training and much better than training on a single site (F1 0.542). If correct, this means hospitals could jointly build phenotyping tools while keeping raw clinical text in place, a practical route toward learning health systems that learn from distributed data.","feed_headline":"Federated clinical phenotyping ties centralized at F1 0.724","feed_subtitle":"Patient notes stay put while a two-stage federated model matches pooled-data phenotyping.","key_machinery":"The load-bearing mechanism is federated averaging, applied twice. At stage 1, a deep averaging network (a network that averages token embeddings and passes them through a dense layer) takes each patient's clinical notes as standardized medical concept identifiers and is trained to predict ICD and CPT codes; local weights from each simulated site are combined as $W_{\\mathrm{ag}} = \\sum_{k} \\frac{n_k}{N} W_k$, with $n_k$ the number of patients at site $k$. The frozen dense layer then maps any new patient's notes to a fixed-length vector. At stage 2, a linear-kernel support vector machine is trained on those vectors to classify each of the sixteen obesity-related phenotypes, again through local training plus sample-size-weighted averaging. The central identity the argument rests on is this weighted averaging update: the claim is that it preserves almost all of the accuracy of centralized training while letting each site keep its raw data.","core_discovery":"The central discovery is that federated averaging, applied at both stages, preserves the accuracy of centralized training while keeping data in place. In the authors' experiments, a deep averaging network pre-trained to predict ICD and CPT codes from standardized medical concept identifiers, with weights averaged across ten simulated sites, produced patient representations that supported a federated linear-kernel support vector machine phenotype classifier across three simulated sites. The fully federated configuration achieved F1 0.724, which the paper states was not statistically different from the centralized two-stage baseline (experiment 4, F1 0.726 in Table 2), and exceeded single-source phenotyping (F1 0.542). The paper also reports that federated phenotyping without representation learning (F1 0.632) matched centralized phenotyping without representation learning (F1 0.634), showing the federated mechanism works at both stages independently.","pith_inferences":["A natural next step is to test the pipeline on naturally partitioned data, such as notes from different hospital systems with different demographics and documentation styles; the random-split simulation likely understates real-world heterogeneity.","If site distributions differ enough, sample-size-weighted averaging may need per-site regularization or personalization, a possibility the paper does not explore.","Combining this scheme with cryptographic aggregation or differential privacy could strengthen the privacy story, but the paper does not measure the resulting communication or accuracy costs.","Per-disease results vary widely (for example, gout F1 0.938 versus GERD F1 0.558), so the averaged gain may not hold uniformly; identifying which phenotypes benefit most would sharpen the claim."],"forward_implications":["Multi-site phenotyping becomes feasible without a central data repository: each site trains locally and only model weights are exchanged.","Federated representation learning on large general corpora can transfer to smaller task-specific annotated sets, raising phenotype accuracy from 0.542 (single-site) to 0.724 (fully federated).","Because stage 1 notes do not need target-phenotype labels, hospitals can contribute routine notes to representation learning without extra annotation effort.","The same two-stage recipe applies to raw text or other feature representations, as the authors state, so it is not tied to standardized concept identifiers."],"supporting_citations":[{"why":"Supplies the deep averaging network architecture and the transfer-learning baseline that the two-stage federated pipeline extends.","marker":"Dligach and Miller, 2018"},{"why":"Provides the federated averaging algorithm for training deep networks on decentralized data, used in both stages.","marker":"McMahan et al., 2016"},{"why":"Gives the communication-efficient federated optimization framework and the sample-size-weighted averaging used for aggregation.","marker":"Konečný et al., 2016"},{"why":"The MIMIC-III corpus is the data source for federated patient representation learning in stage 1.","marker":"Johnson et al., 2016"},{"why":"The i2b2 obesity challenge dataset and its annotations define the phenotyping task and the 16 obesity-related phenotypes.","marker":"Uzuner, 2009"}],"fun_headline_variants":["Federated two-stage phenotyping matches centralized accuracy","Federated model keeps notes local, matches centralized F1","Two-stage federated NLP phenotyping matches pooled-data result","Federated averaging preserves accuracy across both stages","Federated phenotyping hits F1 0.724, matching centralized"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the simulated silos, created by randomly splitting patients, behave like real hospitals; if actual hospitals differ systematically in population, note style, or coding practice, the reported parity with centralized training may not survive.","fun_headline_variants_meta":{"raw":{"variants":["Federated two-stage phenotyping matches centralized accuracy","Federated model keeps notes local, matches centralized F1","Two-stage federated NLP phenotyping matches pooled-data result","Federated averaging preserves accuracy across both stages","Federated phenotyping hits F1 0.724, matching centralized"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000498,"raw_usage":{"total_tokens":2399,"prompt_tokens":861,"completion_tokens":1538,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":477,"completion_tokens_details":{"reasoning_tokens":1455}},"tokens_in":477,"tokens_out":1538,"duration_ms":11410,"temperature":1.0,"reasoning_tokens":1455,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T13:21:33.912701+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same two-stage pipeline with uneven, non-identical data across sites (for instance, split the i2b2 discharge summaries by institution or note style, or split the representation-learning corpus by admission year) and compare the federated F1 with centralized and single-site F1. If the federated F1 falls toward the single-site level, the paper's claim of parity with centralized training is refuted.","supporting_citations":[{"cited_title":"Learning Patient Representations from Text","cited_arxiv_id":"1805.02096","evidence_quote":"Supplies the deep averaging network architecture and the transfer-learning baseline that the two-stage federated pipeline extends."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The MIMIC-III corpus is the data source for federated patient representation learning in stage 1."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The i2b2 obesity challenge dataset and its annotations define the phenotyping task and the 16 obesity-related phenotypes."}],"review_version":1}