REVIEW 4 major objections 5 minor 2 references
Predicting Early-Onset Colorectal Cancer with Large Language Models
T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read A fine-tuned large language model reading electronic health record data from months before diagnosis predicts early-onset colorectal cancer with 73% sensitivity and 91% specificity, performance the authors describe as comparable to faecal…
desk verdict The paper reports a genuine benchmark of fine-tuned GPT-4o for EoCRC prediction, but the LLM-vs-ML comparison is confounded by guideline injection and the prediction window is symptomatic, so the FIT-equivalence claim does not hold. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is a five-part prompt that turns a patient's recent coded EHR history into a narrative input for a fine-tuned large language model: a role definition, chain-of-thought reasoning steps, general CRC assessment guidelines synthesized from national screening recommendations, a fixed output format, and the patient-specific data (coded conditions, lab results with units, and observations, with only the latest value of each lab). The mechanism works by letting the model reason over the longitudinal pattern of codes in plain language, using the medical knowledge embedded in the base model plus the fine-tuning signal from 1,853 CRC and 1,853 non-CRC training patients; it deliberately excludes the month immediately before diagnosis to avoid leakage. For the statistical ML baselines, the paper uses Youden's J index to set decision thresholds under 1% CRC prevalence and aligns feature spaces across training and test runs.
What would settle it
Re-run the evaluation on all 18-to-44-year-olds in the same EHR systems without requiring seven months of prior data, and measure sensitivity separately for patients whose first visit was within a month of diagnosis and for patients with sparse visits; if the model misses most of those, the reported 73% sensitivity is an artifact of the eligibility filter.
Extended reading notes
Core claim
On its own terms, the paper's discovery is that a fine-tuned GPT-4o, prompted with a patient's coded conditions, lab results, and observations from the window 2–7 months before diagnosis, can identify future early-onset colorectal cancer cases at 73% average sensitivity and 91% average specificity, with 99.7% negative predictive value, in a test setting with 1% CRC prevalence. The authors also find that the base GPT-4o, without any fine-tuning, already performs comparably to random forest on the sensitivity-specificity trade-off, and that supervised fine-tuning improves recall by more than 10 percentage points. The LLM achieves this without explicit feature engineering, and the paper reports that its performance is comparable to faecal immunochemical tests and 34% higher in sensitivity than guaiac faecal occult blood tests.
Load-bearing premise
The load-bearing premise is that EoCRC patients with at least seven months of prior EHR data fairly represent the entire 18–44 screening population, even though the paper itself shows that many patients were diagnosed within a month of their first visit and that 6% of cases were pregnant women whose prenatal visits created the required history.
Editorial extensions
If this is right
- If the 73%/91% result holds in a real screening population, an EHR-based LLM flag could serve as a rule-out test: patients the model clears have a 99.7% chance of not being diagnosed with EoCRC in the study's window.
- The base GPT-4o, with no fine-tuning, produced a sensitivity/specificity balance close to random forest, suggesting that conversational prompting alone can extract predictive signal from structured coded data.
- Because precision is only 7.6% at 1% prevalence, the model's practical role would be to prioritize screening resources rather than to diagnose; most positive flags will be false positives.
- The reported 'comparable to FIT' performance means an EHR-only risk flag could offer a screening decision aid where stool-based tests are not available or not yet scheduled.
- The model's natural-language explanations may be easier for clinicians and patients to interpret than conventional feature-importance or instance-attribution explanations.
Reading between the lines
- My inference: if the seven-month eligibility filter were removed, sensitivity would likely fall, because the paper's own exclusion data show that a substantial share of young CRC patients enter care only near diagnosis; a valuable next step is evaluating the model on 'first-visit-near-diagnosis' patients with a sparse-history input mode.
- My inference: the 6% pregnancy over-representation means the control sampling strategy matters; comparing CRC patients primarily against randomly sampled non-CRC patients may let the model learn 'pregnancy with prenatal visits' as a proxy for healthcare contact rather than for cancer risk. A sex- and visit-frequency-matched control design would test this.
- My inference: because the LLM emitted probabilities only as discrete Likert-like values, ROC analysis was impossible; prompting the model to output a continuous risk score (or extracting token logits) would let clinicians tune thresholds and compare directly with FIT calibration curves.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The manuscript reports a retrospective case-control study using the Truveta multi-health-system EHR database to predict early-onset colorectal cancer (EoCRC) in patients aged 18–44. The authors constructed a cohort of 1,953 CRC patients and sampled non-CRC controls, trained 10 statistical machine learning models and a fine-tuned GPT-4o model on features from months 2–7 before diagnosis, and evaluated on 10 test runs with 1% CRC prevalence. The fine-tuned GPT-4o achieved 73% average sensitivity and 91% average specificity, with a 99.7% negative predictive value, and the authors claim this performance is comparable to faecal immunochemical tests. The paper also presents XGBoost feature importance, SHAP explanations, and a natural-language explanation from the LLM.
Significance. If the results are robust, the study would be a useful addition to the literature on LLM-based risk prediction from EHR data, and it is one of the first to target the under-45 age group using multi-site U.S. data. The authors make several commendable choices: a held-out test set with realistic 1% prevalence, ten ML models as a broad baseline, reporting of confidence intervals, and an explainability section. However, the central claim that LLMs outperform traditional ML is confounded by the fact that the LLM prompt contains USPSTF screening guidelines and chain-of-thought instructions that the ML models do not receive, and the claim of comparability to FITs is not supported by the symptomatic, pre-diagnostic window used for evaluation. These issues prevent the manuscript from being acceptable in its current form.
major comments (4)
- [Methods, 'Large Language Models' and 'Statistical Machine Learning models'] The LLM prompt includes component 2 (Chain-of-Thought reasoning steps for CRC determination) and component 3 (general guidelines for CRC diagnosis synthesized from USPSTF recommendations), as described in the Large Language Models section and shown in Figure 1. The ML models receive only the engineered feature vectors described in the Statistical Machine Learning models section. The reported performance difference is therefore not attributable to the model family alone; it may reflect the injection of external domain knowledge into the LLM. To support the authors' conclusion that fine-tuned GPT-4o is a better predictor, the comparison needs an ablation: either remove the guidelines and chain-of-thought from the LLM prompt, or provide the ML models with equivalent guideline-derived features. Without this, the comparison in Table 3 is confounded.
- [Results, Table 3; Conclusion] The claim in the Conclusion that the fine-tuned LLM's performance is 'comparable to faecal immunochemical tests (FITs)' is not supported by the evaluation design. The prediction window is months 2–7 before diagnosis, and no exclusion or stratification of symptomatic patients is described. Figure 2 shows that 'fresh blood passed per rectum' and 'carcinoembryonic lab test' are among the top XGBoost features, indicating that symptom signals are present in the window. FITs are validated as screening tests in asymptomatic average-risk populations, whereas the test cohort here is enriched to 1% CRC prevalence and restricted to patients with at least 7 months of prior EHR activity. The reported 73% sensitivity may partly reflect detection of symptoms that would trigger diagnostic workup in routine care. The authors should either restrict the evaluation to patients without CRC-related symptoms in the window, or limit the claim to a statement of sensitivity/specificity in a pre-diagnostic clinical context rather than comparability to screening FITs.
- [Methods, 'Dataset'] The control group's index date is not defined. The CRC group's features are defined relative to the diagnosis date (months 2–7 prior). Since non-CRC controls have no diagnosis date, it is unclear what reference date was used to define their 7-month observation window and to ensure they had comparable data density. If controls were sampled without requiring the same 7 months of clinical history, the model could use the absence of data as a non-CRC signal, inflating specificity. The Methods paragraph stating that 'any patients without 7 months of medical conditions, lab results and observations would be excluded' needs to specify how this was applied to controls, and the authors should report the distribution of observation-window lengths in both groups.
- [Methods, 'Large Language Models'; Results, Table 3] The decision rule for the LLM is unspecified. The authors report that the model was asked for a probability score but that the scores were on a Likert scale and therefore not usable for an ROC curve. However, the paper does not state how the binary prediction used to compute the Table 3 sensitivity and specificity was derived from the model output (e.g., parsing of the specified output format, any threshold on the Likert score, or majority voting across runs). For the ML models, the threshold was explicitly chosen via Youden's J with cross-validation. The LLM evaluation needs an equally explicit and reproducible decision rule so that the headline 73%/91% figures can be verified.
minor comments (5)
- [Throughout] The manuscript text contains numerous OCR-like artifacts (e.g., 'Predic'ng' in the title, 'Sena' for 'Seattle', 'agempted' for 'attempted', 'Sensi.vity' for 'Sensitivity'); a careful proofread is needed before publication.
- [Figure 1] The prompt template shown in Figure 1 is referenced but not actually reproduced in the visible manuscript; since the prompt is central to the LLM evaluation, the full template should be included in an appendix.
- [Results, Table 3] The paper calls the 99.7% figure 'the accuracy of identifying the Non-CRC cases', but this value is the negative predictive value, not accuracy; the wording should be corrected.
- [Results, Table 3] In Table 3, the LightGBM and HGBoost rows have extremely low specificity (0.084 and 0.005) with very large confidence intervals on NPV; presenting per-run distributions or median values would be informative, since the average over 10 runs may be dominated by a few degenerate runs.
- [Data availability] The manuscript does not include a data or code availability statement; for a study claiming a methodological comparison, providing the prompt template, fine-tuning configuration, and feature engineering pipeline would be valuable for reproducibility.
Circularity Check
No circularity: the prediction task is a standard supervised classification with held-out test data, and the reported sensitivities and specificities are empirical results rather than consequences of the input construction.
full rationale
The paper's central claim is an empirical performance comparison, not a derivation that reduces to its inputs. The Methods describe a standard train/test split: 1,853 CRC patients and 1,853 non-CRC patients form the training set, and 100 CRC patients are held out and evaluated in 10 runs mixed with 990 non-CRC patients per run to mimic 1% prevalence. The target outcome (CRC diagnosis) is not used in the feature construction or in the LLM prompt; the prompt includes only patient conditions, lab results, and observations from the 2nd to 7th month before diagnosis, explicitly excluding the final month to avoid signals highly indicative of impending diagnosis. The fine-tuned LLM is trained on the same training set and evaluated on held-out data, so the 73% sensitivity and 91% specificity are genuine out-of-sample measures. The USPSTF guideline component in the LLM prompt is an external, citable knowledge source, not a fitted parameter or a label-derived feature. The paper's acknowledged limitations — such as requiring 7 months of EHR history, the 6% of pregnant CRC patients, and the exclusion of patients diagnosed within one month of first visit — concern selection bias and generalizability to a screening population, not circularity. The comparison to FITs is a claim about external benchmark comparability and may be contestable on population-mismatch grounds, but that is a validity or correctness concern, not circular reasoning. There are no load-bearing self-citations or imported uniqueness theorems; the cited prior work (e.g., Zhen et al., USPSTF, Cochrane review) is external. No step in the paper's derivation chain is equivalent to its inputs by construction, and no fitted parameter is renamed as a prediction. Therefore the circularity score is 0.
Assumptions & free parameters
free parameters (1)
- Classification threshold for LLM and ML models =
Youden's J-index optimized via 10-fold CV
assumptions (4)
- domain assumption EHR coded data (ICD, SNOMED-CT, LOINC) accurately reflects the presence of conditions, lab results, and observations.
- domain assumption The USPSTF recommendations provide appropriate guidance for CRC diagnosis and were correctly synthesized in the prompt.
- ad hoc to paper Patients with at least 7 months of prior EHR data are representative of the target EoCRC screening population.
- standard math A balanced training set with under-sampling is an appropriate design for training models on rare outcomes.
Cite this review
Pith. "Pith review of Predicting Early-Onset Colorectal Cancer with Large Language Models." pith.science (2026). https://pith.science/paper/BRJVKHTS
@misc{pith2026250611410,
author = {Pith},
title = {Pith review of: Predicting Early-Onset Colorectal Cancer with Large Language Models},
year = {2026},
howpublished = {\url{https://pith.science/paper/BRJVKHTS}},
note = {Machine review of arXiv:2506.11410}
}
read the original abstract
The incidence rate of early-onset colorectal cancer (EoCRC, age < 45) has increased every year, but this population is younger than the recommended age established by national guidelines for cancer screening. In this paper, we applied 10 different machine learning models to predict EoCRC, and compared their performance with advanced large language models (LLM), using patient conditions, lab results, and observations within 6 months of patient journey prior to the CRC diagnoses. We retrospectively identified 1,953 CRC patients from multiple health systems across the United States. The results demonstrated that the fine-tuned LLM achieved an average of 73% sensitivity and 91% specificity.
Figures
Figures from the paper (1 more)
Reference graph
Works this paper leans on
-
[6]
Chen FW , Sundaram V, Chew TA, Ladabaum U. Advanced-stage colorectal cancer in persons younger than 50 years not associated with longer dura.on of symptoms or .me to diagnosis. Clinical gastroenterology and hepatology. 2017;15(5):728-37. 7. Townsend JS, Jones MC, Jones MN, Waits AW, Konrad K, McCoy NM. A case study of early-onset colorectal cancer: using ...
work page 2017
-
[28]
Universal Abstrac.on: Harnessing Fron.er Models to Structure Real-World Data at Scale; 2025
Wong C, Preston S, Liu Q, Gero Z, Bagga J, Zhang S, et al.. Universal Abstrac.on: Harnessing Fron.er Models to Structure Real-World Data at Scale; 2025. Available from: hgps://arxiv.org/abs/2502.00943. 29. Bousselham H, Mourhir A, et al. Fine-tuning GPT on biomedical NLP tasks: an empirical evalua.on. In: 2024 Interna.onal Conference on Computer, Electric...
arXiv 2025
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.