{"id":"4f40f78b-6414-446b-807e-655db06866c9","arxiv_id":"2412.10848","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Fine-tuning a 7B LLM on contextualized patient timelines from MIMIC-III substantially improves next-concept and one-month risk predictions over prior models.","lead":"Foresight 2 is a 7-billion-parameter language model fine-tuned on hospital notes from MIMIC-III to predict which medical conditions, medications, and procedures will appear next in a patient's record. The authors report large gains over their earlier model and over zero-shot GPT-4-turbo on this forecasting task.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The evaluation labels are MedCAT extractions shared by training and testing, so FS2 may be rewarded for predicting extractor artifacts; a human-validated or ICD-coded label check is needed before the headline P/R and risk-forecast claims can be accepted.","rationale":"The most decisive vulnerability in the paper is the status of the evaluation labels. The whole pipeline — Section 2.1 (MedCAT extraction and timeline construction), Section 2.3 (precision/recall against extracted concepts), and Section 2.4 (risk-forecast labels from extracted future disorders) — treats SNOMED concepts produced by MedCAT as ground truth. Section 4.1 explicitly acknowledges MedCAT is imperfect, but no human-annotated test set or independent label source is supplied. This matters more than whether the FS1-vs-FS2 comparison is internally fair: if MedCAT has systematic biases, FS2's access to surrounding free text lets it learn those biases directly, while FS1 (concept-only input) cannot. The headline numbers could then reflect proficiency at emulating MedCAT rather than clinical forecasting. The risk comparison inherits the same issue and adds a GPT-4-based validator, which is not an independent gold standard. The proposed check — clinician annotation or ICD-code proxy on a sample — would settle whether the concern lands. If the numbers hold, the central claim is substantially supported; if they drop, the paper would need to be reframed as modeling extracted concepts rather than clinical events. There are also secondary concerns about the zero-shot versus fine-tuned comparison and differing support across models in Table 2, but they are secondary to the label-validity issue. I therefore agree with the reader's weakest assumption and do not move the verdict: CONDITIONAL remains the right call.","tokens_in":17624,"tokens_out":8950,"duration_ms":85047,"concrete_test":"Select a random sample of roughly 300 patients from the 2,101-patient test split. Have two clinicians independently annotate, directly from the raw MIMIC-III notes, which new disorders occur within 30 days after the risk-forecast split point for each patient; alternatively, use MIMIC-III's structured ICD-9 discharge codes mapped to SNOMED as an independent proxy. Recompute Table 1 rows 'Disorders, T=30, @1' and Table 2 'At least 1' for FS2-Mistral and GPT-4-turbo against these independent labels, using the same model outputs and support. If FS2's precision/recall or the 90% risk-forecast figure drops by more than about 10 percentage points relative to the MedCAT-label evaluation, the reported improvement is substantially an artifact of label noise; if the numbers hold, the central forecasting claim survives.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 2.1 constructs both training and test timelines by running MedCAT on MIMIC-III notes, and Section 2.3 scores precision/recall against those same extracted labels; Section 4.1 concedes MedCAT is imperfect. Because the same extractor produces training and test labels, a model can inflate its score by learning MedCAT's systematic biases (for example, which surface forms it over-extracts or which concepts it misses in context) rather than by predicting clinically real events. FS2 is especially susceptible because it is trained on the raw context surrounding each extracted concept, so it can exploit textual cues that correlate with MedCAT errors, whereas FS1 saw only extracted concepts. The risk-forecast comparison inherits the same problem: the 'ground truth' new disorders in Table 2 are also MedCAT extractions from the future month, and the baseline models are evaluated against these labels via a GPT-4 validator rather than against human-annotated outcomes. If MedCAT's errors are systematic, both the absolute P/R values (0.73/0.66, 0.69/0.62) and the risk-forecast gap (90% vs 65%) overstate real-world forecasting accuracy. The paper provides no human-annotated validation set and no independent label source, so the central claim that FS2 forecasts future clinical events is not yet established.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents Foresight 2 (FS2), a 7B-parameter LLM (Mistral-v0.1 and LLaMA-2 variants) fine-tuned on MIMIC-III free text to model patient timelines. The timeline is built by running MedCAT to extract SNOMED concepts, retaining surrounding context sentences, bucketing by day, filtering singleton concepts, and reconstructing a single clinical note in which concepts are replaced by SNOMED tokens. FS2 is trained with a modified language-modeling objective that computes loss only on concept tokens. The paper reports large improvements over the prior FS1 model for next-new-concept prediction (P/R 0.73/0.66 vs 0.52/0.32 for all concepts; 0.69/0.62 vs 0.46/0.25 for disorders) and a risk-forecasting result in which FS2-Mistral gets at least one correct top-5 prediction in 90% of patients versus 65% for GPT-4-turbo. The authors also describe a tokenizer-embedding initialization for SNOMED codes, a risk-forecasting second-stage fine-tune, and a GPT-4-based automated validation pipeline.","tokens_in":17849,"tokens_out":5673,"duration_ms":50499,"significance":"If the headline results are trustworthy, the paper makes a valuable empirical contribution: it shows that a small open-weight LLM fine-tuned on real hospital timelines can substantially outperform both a purpose-built timeline transformer (FS1) and much larger zero-shot generalist LLMs on next-concept and short-term risk prediction. The methodological ideas of contextualized timelines, SNOMED-as-tokens, and concept-only loss are practical and reusable. The main caveat is that the evaluation labels are generated by the same MedCAT extraction pipeline that defines the input, so the absolute and relative improvements could partly reflect learning the extractor's systematic behavior rather than learning to forecast clinically real events. The risk-forecast comparison is also between a task-fine-tuned model and zero-shot baselines, which conflates fine-tuning with model quality. The paper is transparent about several limitations, including MedCAT imperfection and the need for external validation.","major_comments":[{"comment":"The training and test labels for both the next-concept task and the risk-forecast task are SNOMED concepts extracted by MedCAT from MIMIC-III notes, with no human-annotated or independent label source. Section 2.1 states that MedCAT produces the concepts used to build the timelines, Section 2.3 defines precision/recall against these extracted labels, and Section 4.1 concedes that MedCAT is imperfect. Since the same extractor defines both input and ground truth, the reported P/R values (e.g., 0.73/0.66 for all new concepts) may reflect the model learning MedCAT's systematic errors (e.g., over-extraction of certain surface forms, missed concepts in context) rather than predicting true clinical events. This is a load-bearing issue for the central claim. The authors should provide a human-validated subset of the test labels (or an independent label source such as MIMIC-III ICD-coded diagnoses) and report metrics on that subset, or otherwise quantify the agreement between MedCAT-extracted labels and a reference standard.","section":"§2.1, §2.3, §4.1"},{"comment":"The risk-forecasting comparison is not apples-to-apples: FS2 is explicitly fine-tuned on the exact task (predicting new disorders in the month after a timeline split), whereas GPT-4-turbo, BioMistral, MedAlpaca, and MEDITRON are evaluated zero-shot with prompts. The reported gap (90% vs 65% for at least one correct top-5 prediction) therefore conflates task-specific fine-tuning with model capability. In addition, the support varies across models (e.g., GPT-4-turbo only 472/535 patients, BioMistral 288/535), and no confidence intervals or significance tests are reported. The authors should either fine-tune the baseline models on the same risk-forecasting task under comparable conditions, or clearly frame the result as 'fine-tuned specialized model vs zero-shot generalist' and discuss the support mismatch. At minimum, they should report refusal rates and characteristics of excluded patients.","section":"§2.4, Table 2"},{"comment":"The risk-prediction outputs are validated automatically by GPT-4-turbo using a prompt that explicitly allows synonyms and 'similar' disorders to count as correct. The manual check described is limited to confirming that the validator 'makes sense,' not a systematic clinician annotation of the full 535-patient test set. Combined with the fact that the ground-truth disorders are also MedCAT extractions from the subsequent month, this validation pipeline can both over-approximate true matches and perpetuate label noise. The authors should provide a human-annotated sample of the risk outcomes (e.g., 100–200 patients) and report agreement between the GPT-4 validator and clinician labels, or otherwise validate the risk-forecast numbers against an independent outcome source.","section":"Appendix A.1, §2.4"}],"minor_comments":[{"comment":"The privacy claim that the model 'was not directly trained on text' is contradicted by Section 2.1, where contextual free text is part of the input, and by Section 2.2, where the model can generate any token in the vocabulary. The paper should correct this statement or clarify that loss is only computed on concept tokens, not that the model never sees text.","section":"§4"},{"comment":"The support columns 'Sup N' and 'Sup R' are not clearly tied to the model columns; the reader must infer that the support is the same across FS1 and FS2 for each row. Please add a sentence or footnote clarifying what these numbers represent and why they are identical across models.","section":"Table 1"},{"comment":"The 'Ground Truth' column is described as coming from the EHR, but it is presumably also MedCAT-derived from the notes. Please state whether these examples were human-validated, since the table is meant to illustrate qualitative performance.","section":"Table 4"},{"comment":"For GPT-4-turbo risk prediction, the paper does not specify how the patient history is truncated if it exceeds the model's context window, or why 63 of 535 patients are marked as refusals. Please clarify the inclusion criteria and the token-count handling.","section":"Appendix A.2"},{"comment":"Several references use placeholder 'et al.' forms (e.g., 'OpenAI et al. 2023b', 'Gemini Team et al. 2023a'), which should be resolved to full author lists or handled consistently with the journal's citation style.","section":"References"},{"comment":"The GitHub link is removed for anonymity; for reproducibility, the final version should either include the repository link or provide a detailed supplement describing training configurations, data preprocessing, and evaluation scripts.","section":"Abstract and §1"}],"recommendation":"major_revision","confidential_remarks":"The paper has a plausible and interesting empirical result, and the authors are transparent about many limitations. However, the central claim is currently undermined by the absence of any human-validated or independent test labels, and the risk-forecast comparison is not controlled for fine-tuning. These are fixable within the scope of the manuscript: a modest human-annotation effort (e.g., a few hundred patients) or an ICD-based label check would substantially raise confidence. The comparison against GPT-4-turbo should also be reframed or rerun with a tuned baseline. I see no reason to reject at this stage, but the requested validation is necessary before the headline numbers can be accepted."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Read the FS2 paper. The short version: the modeling contribution is real, the evaluation is only partly convincing, and the risk comparison in particular overstates the case.\n\nWhat's actually new: they fine-tune a 7B LLM (Mistral or LLaMA) on patient timelines where each SNOMED concept keeps its surrounding sentence as context, add the SNOMED codes as single tokens with embeddings initialized from the concept name, and then do a second-stage fine-tune for one-month risk. The context ablation is the most convincing piece—drop the context and performance falls by roughly 40%, which shows the design matters. The comparison against FS1 uses the same custom metrics and pipeline, so the relative improvement (0.73/0.66 vs 0.52/0.32 for next-new-concept) is plausible.\n\nThe soft spot is the label source. Both training and test timelines are built with MedCAT extractions, and precision/recall are measured against those same labels. MedCAT is a good extractor (F1 over 0.9), but if it has systematic biases—over-extracting certain surface forms, missing concepts in context—the model can learn those artifacts. The absolute numbers are accordingly optimistic. A human-validated sample of test labels, or a comparison against ICD-coded outcomes, would settle this. The paper admits MedCAT is imperfect in Section 4.1, but does not provide that validation.\n\nThe risk-forecast comparison is weaker. FS2 was fine-tuned on the task; the baselines were zero-shot with different prompts. Support varies (535, 472, 288...), and the validation uses GPT-4-turbo as the judge, with a clinician checking the validator rather than the labels. That 90% vs 65% is a system-comparison result, not a clinical accuracy claim. I would not quote it as evidence of clinical readiness.\n\nSmaller issues: no code or data artifacts released (the GitHub link is removed), no confidence intervals, and the random patient split means the test set includes the same time period as training, which may inflate results if note styles drift.\n\nWho should read this: clinical NLP and informatics researchers working on EHR timeline models. They'll get a solid recipe and a new baseline, and they'll know to re-validate the labels before trusting the numbers.\n\nRecommendation: send it to peer review. It is a serious empirical system paper with an honest limitations section; the label and baseline issues are fixable with additional experiments and should be raised by reviewers, not used to desk-reject.","headline":"A credible system paper on contextualized SNOMED timelines for clinical forecasting, but MedCAT-derived labels and uneven risk baselines keep the headline numbers from being trusted as clinical accuracy.","tokens_in":18447,"tokens_out":3007,"would_cite":false,"duration_ms":28213,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Foresight 2, a 7-billion-parameter model fine-tuned on hospital notes, predicts future SNOMED concepts and one-month disorder risk substantially better than previous timeline models and zero-shot general LLMs.","keywords":["patient timelines","medical forecasting","SNOMED CT","MIMIC-III","fine-tuned LLMs","electronic health records","next concept prediction","risk forecasting"],"falsifier":"Take a random sample of the test-set patients, have clinicians manually annotate the true next new concepts and next new disorders in the relevant follow-up windows, recompute precision and recall for FS2 and FS1 on that subsample, and compare with the paper's numbers; if the gap collapses or reverses, the central claim is falsified.","tokens_in":17352,"feed_emoji":"🩺","tokens_out":12798,"duration_ms":94879,"temperature":0.7,"pith_summary":"The paper tries to establish that a compact language model, fine-tuned on hospital free text organised into contextualised patient timelines, can forecast what will happen next in a patient's record. It reports precision/recall of 0.73/0.66 for the next new biomedical concept and 0.69/0.62 for the next new disorder, against 0.52/0.32 and 0.46/0.25 for the previous state of the art. On a one-month risk-forecast task, FS2 puts at least one correct disorder in the top five for 90% of patients, while GPT-4-turbo manages 65%. If correct, the result shows that specialised hospital data can outperform much larger general models on real clinical forecasting, and that a structured ontology such as SNOMED can serve as the model's output space.","feed_headline":"7B patient-history model beats GPT-4 on next-month risk","feed_subtitle":"FS2 fine-tuned on MIMIC-III timelines beats GPT-4-turbo 90% to 65% on next-month disorder risk.","key_machinery":"The central object is the contextualised patient timeline: a chronological per-patient sequence in which each extracted SNOMED concept is represented by its code as a single token, surrounded by the sentence or token window where it appeared, with age, sex, ethnicity, and temporal separators such as <7 days later> inserted between distant events. The mechanism carrying the argument is the supervised fine-tuning objective that predicts only concept tokens at concept positions, which forces the model to use the clinical context while keeping the output vocabulary inside SNOMED. A second-stage fine-tuning task splits each timeline and trains the model to predict the unique new disorders appearing in the first month after the split, aligning the evaluation with a 30-day horizon.","core_discovery":"Foresight 2 (FS2) is a fine-tuned LLM that models a patient's history as a sequence of biomedical concept tokens interleaved with the free-text context in which those concepts were mentioned. It is trained with a modified language-modelling objective: loss is computed only on SNOMED concept tokens at concept positions, never on the surrounding words, and each SNOMED code is added to the tokenizer as a single token whose embedding starts from the average of the embeddings of the words in its name. The paper reports that this contextualised-timeline setup yields large gains over Foresight 1, which saw only bare concept sequences, and that after a second fine-tuning stage FS2 predicts new disorders within the next month better than GPT-4-turbo, BioMistral, MedAlpaca, and MEDITRON. The authors' central conclusion is that incorporating real hospital data into LLMs matters more than model scale, and that standardised ontologies make the predictions usable in clinical information systems.","pith_inferences":["Editorial inference: the paper does not run a human-annotated validation of the test labels, so the most direct check on its numbers would be a clinician-annotated subsample; without it, the reported gains could partly reflect agreement with MedCAT's extraction patterns rather than true clinical events.","Editorial inference: the 40% ablation drop suggests that extending the retained context beyond sentence-width windows, or attending over the full note, might push accuracy further; the paper tests only the sentence-window setting.","Editorial inference: because the output vocabulary is limited to SNOMED, genuinely novel or undocumented conditions cannot be predicted by design, so the real-world ceiling depends on ontology coverage as much as on model skill.","Editorial inference: the comparison with GPT-4-turbo uses different effective test sets, since some baseline models skipped long sequences or refused to answer, so a like-for-like head-to-head on exactly the same 535 patients would sharpen the 90% versus 65% claim."],"forward_implications":["For next-new-concept prediction, FS2-Mistral more than doubles FS1's recall while also improving precision, reaching the high-precision regime needed for alerting systems that try to avoid clinician alert fatigue.","Removing the surrounding context from timelines drops performance by about 40%, so the free-text context is a major carrier of predictive signal rather than a cosmetic addition.","Because the model outputs SNOMED codes, predictions are standardised, can be ranked by probability, and need no separate mapping step to integrate with existing EHR terminology systems.","The one-month risk-forecast horizon matches operational targets such as reducing 30-day readmissions, giving the model a concrete deployment use case."],"supporting_citations":[{"why":"It defines Foresight 1, the previous state-of-the-art baseline and the patient-timeline pipeline that FS2 builds on.","marker":"Kraljevic et al. [2023]"},{"why":"It provides MedCAT, the concept-extraction tool whose SNOMED annotations become the timelines and the evaluation labels.","marker":"Kraljevic et al. [2020]"},{"why":"It supplies MIMIC-III, the hospital free-text dataset used for training and testing.","marker":"Johnson et al. [2016]"},{"why":"It provides Mistral-7B, one of the two pretrained base LLMs that FS2 fine-tunes.","marker":"Jiang et al. [2023]"},{"why":"It provides LLaMA-2-7B, the other pretrained base LLM used in the FS2-LLaMA variant.","marker":"Touvron et al. [2023b]"},{"why":"It is GPT-4-turbo, the zero-shot general model that serves as the main risk-forecasting comparison point.","marker":"[et al., 2023b]"},{"why":"It contributes the filtering step that removes concepts mentioned only once as probable extraction errors.","marker":"Bean et al. [2023]"},{"why":"It defines SNOMED-CT, the ontology used as the token-level vocabulary and output space.","marker":"Stearns et al. [2001]"}],"fun_headline_variants":["Small model, huge win: FS2 beats GPT-4 on patient risk","Hospital-trained LLM predicts next-month disorders better than GPT-4","FS2: fine-tuned 7B model outperforms GPT-4 in medical forecasting","Patient timelines beat scale: FS2 tops GPT-4 in risk prediction","SNOMED-aware LLM from MIMIC-III surpasses GPT-4 for next-month diagnoses"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that MedCAT's automatically extracted SNOMED concepts are accurate enough to serve as the correct answers for both training and testing; the paper's Section 4.1 admits MedCAT is imperfect, so if extraction errors are systematic, the reported precision and recall may measure agreement with the extractor rather than with true clinical events.","fun_headline_variants_meta":{"raw":{"variants":["Small model, huge win: FS2 beats GPT-4 on patient risk","Hospital-trained LLM predicts next-month disorders better than GPT-4","FS2: fine-tuned 7B model outperforms GPT-4 in medical forecasting","Patient timelines beat scale: FS2 tops GPT-4 in risk prediction","SNOMED-aware LLM from MIMIC-III surpasses GPT-4 for next-month diagnoses"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000776,"raw_usage":{"total_tokens":3459,"prompt_tokens":1002,"completion_tokens":2457,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":618,"completion_tokens_details":{"reasoning_tokens":2351}},"tokens_in":618,"tokens_out":2457,"duration_ms":15709,"temperature":1.0,"reasoning_tokens":2351,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T15:33:51.727241+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random sample of the test-set patients, have clinicians manually annotate the true next new concepts and next new disorders in the relevant follow-up windows, recompute precision and recall for FS2 and FS1 on that subsample, and compare with the paper's numbers; if the gap collapses or reverses, the central claim is falsified.","supporting_citations":[{"cited_title":"Foresight -- generative pretrained transformer (gpt) for modelling of patient timelines using ehrs","cited_arxiv_id":null,"evidence_quote":"It defines Foresight 1, the previous state-of-the-art baseline and the patient-timeline pipeline that FS2 builds on."},{"cited_title":"Foresight -- generative pretrained transformer (gpt) for modelling of patient timelines using ehrs","cited_arxiv_id":null,"evidence_quote":"It contributes the filtering step that removes concepts mentioned only once as probable extraction errors."}],"review_version":1}