Validation of a CT-brain analysis tool for measuring global cortical atrophy in older patient cohorts

Emma Colbourne; Jasmine Gan; Jim Davies; Ludovica Griffanti; Mark Jenkinson; Nele Demeyere; Sarah T Pendlebury; Sukhdeep Bal; Taylor Hanayik

REVIEW 3 major objections 5 minor 2 references

Validation of a CT-brain analysis tool for measuring global cortical atrophy in older patient cohorts

T0 review · 3 major / 5 minor · reviewed 2026-08-04 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper validates a fully automated deep-learning tool that measures the Global Cortical Atrophy score on routine CT-brain scans in about four seconds, with error against trained human raters comparable to or smaller than the disagreemen

desk verdict Useful proof-of-concept for automated CT-based GCA scoring, but the headline MAE/kappa appear to be computed on the full dataset rather than the held-out test set, and the single-rater reference standard has fair human-human agreement; both need to be fixed before the accuracy claim holds. read the letter →

arxiv 2509.08012 v1 pith:I2RGP5QH submitted 2025-09-08 eess.IV cs.AIcs.CV

Sukhdeep Bal , Emma Colbourne , Jasmine Gan , Ludovica Griffanti , Taylor Hanayik , Nele Demeyere , Jim Davies , Sarah T Pendlebury

show 1 more author

Mark Jenkinson

This is my paper · ORCID

classification eess.IVcs.AIcs.CV

keywords globalcorticalatrophyGCAscaledeeplearningCTbrainimagingautomatedmeasurementvisualratingvalidationolderpatientscognition

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper asks whether a deep-learning tool can replace slow, subjective visual rating of brain atrophy on CT scans with a fast, automatic number that clinicians and researchers can trust. It validates a tool that predicts the Global Cortical Atrophy (GCA) score, a 0-39 visual scale, directly from routine CT images of patients over 65, without any user input. Across 864 real-world scans from acute medicine and stroke patients, the tool's mean absolute error against the primary rater was 3.2 points, its agreement was moderate (weighted kappa 0.45), and half of its predictions fell within two points of the rater. Crucially, the tool agreed with human raters at least as well as the two human raters agreed with each other (kappa 0.28), and it produced each score in four seconds versus about three minutes by hand. If these results hold, the tool would make standardised atrophy measurement feasible at scale on the world's most common brain imaging modality.

What carries the argument

The central object is the GCA score, a visual rating that sums 0-3 severity grades across 13 brain regions (sulcal widening in the frontal, temporal, and parieto-occipital lobes of each hemisphere, plus dilatation of the frontal, occipital, and temporal horns and the third ventricle) into a total out of 39. The tool is a deep-learning regressor that maps pre-processed, skull-stripped, registration-normalised 3D CT volumes directly to this score, bypassing tissue segmentation. The authors operationalised the visual scale for CT with explicit written criteria and reference baseline scans, then trained on 518 scans, optimised on 173, and tested on 173 drawn from 864 routine clinical CTs, with r

What would settle it

Have three or more independent experts rate a common set of, say, 100 CT scans and compare the tool's scores against their consensus; if the tool's weighted kappa against the consensus is no better than the inter-expert kappa, the claim of human-level measurement fails. A cheaper check: retest on scans with GCA below 3 or above 28, where the paper already reports systematic over- and under-estimation.

Watch

Extended reading notes

Core claim

On its own terms, the central claim is that a deep-learning model trained on CT images labelled with one expert rater's visual GCA scores can reproduce those scores accurately enough for research and clinical use in older patient cohorts. The validation reports a mean absolute error of 3.2 between tool and rater-1, a mean signed difference of 0.18 with limits of agreement from -7.8 to 8.2, and Cohen's weighted kappa of 0.45 against rater-1 and 0.41 against a second rater, compared with 0.28 between the two raters. About half of the tool's predictions fell within two GCA points of rater-1. The tool also classified scans into no/mild, moderate, and severe atrophy with 70-73% accuracy against r

Load-bearing premise

The load-bearing premise is that a single rater's visual GCA scores are a valid, reliable ground truth for brain atrophy; if that rater is idiosyncratic, the tool's low error against that rater does not prove it measures atrophy accurately.

Editorial extensions

If this is right

Routine CT scans already in clinical archives could be automatically re-scored at four seconds per scan, making large-scale atrophy phenotyping practical for health-data research.
The numeric GCA output can be plugged into electronic-health-record prediction algorithms alongside clinical variables to flag patients at risk of dementia, delirium, falls, or functional decline.
At point of care, a clinically approved version could add a standardised atrophy score to every CT head report without adding radiologist workload.
Because the tool matches or exceeds human-human agreement, it could act as a consistent reference in studies where multiple expert raters are not available.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Because the model was trained on a single rater's scores as ground truth, its agreement with that same rater partly reflects imitation; a sharper test would compare the tool against consensus labels from several experts, using the same data.
The fact that tool-rater kappa (0.45, 0.41) exceeded human-human kappa (0.28) suggests the tool may be more consistent than individual raters; if replicated, it could serve as an objective bridge between raters and across cohorts.
The reported tendency to under-rate very high GCA scores and over-rate very low ones implies the tool is most dependable in the moderate range; enriching training with extreme scans is a testable extension the authors themselves flag.
A natural next validation is longitudinal: whether the tool can detect within-patient change in GCA over time, since clinical monitoring would depend on change rather than a single absolute score; the paper does not report test-retest or follow-up sensitivity.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The short version: this is a practical, real-world validation of a CT-only deep learning tool that directly outputs the full 0-39 GCA score, which prior work hasn't done. The four-second runtime and the agreement numbers are attractive. But the paper as written does not let you verify the headline accuracy on the held-out test set, and the reference standard is one rater whose own inter-rater agreement is only fair. These are fixable, but they are load-bearing.

What's genuinely new: ref 20 needed paired MRI and used a compressed 0-3 scale; this tool trains directly on CT with the full GCA scale on 864 real clinical scans from acute medicine and stroke. The supplement's operationalization of GCA on CT is a useful contribution on its own—it turns a vague visual scale into something reproducible. The tool-versus-human agreement (kappa 0.45 vs rater-1, 0.41 vs rater-2) compares favorably to human-human agreement (0.28). The Bland-Altman bias is near zero, and the correlations with age and cognition point the expected way. The authors are also upfront in the Discussion that training on a single rater means the model likely reflects that rater's systematic errors.

The soft spots, in proportion. The biggest one: the Results say "Among 864 patients ... MAE ... was 3.2 for the dataset overall." That reads as computed on all 864, including training and optimisation scans. The Methods say testing was on the 173-scan test set, but the test-set MAE and kappa are never reported. The only test-set number is the error distribution (50% within ±2). Until the held-out MAE/kappa are reported, the "measured accurately" claim rests on in-sample fit. That is the difference between generalization and memory, and the stress-test note is right to put it front and center.

Second, the reference standard. Rater-1's labels are both the training target and the test reference. With kappa 0.28 between rater-1 and rater-2 on 20 scans, the human "truth" is not stable. The paper acknowledges this, but the abstract's "accurately" is too strong given that reference. The rater-2 comparison helps, but the paper doesn't say whether those 20 scans were in the training/optimisation split, which matters for interpreting kappa 0.41.

Third, no code, architecture, or data release, so no independent replication. For a validation study that's a real constraint, though not a reason to reject.

Who is this for? Researchers working on CT-based biomarkers or clinical AI validation. A careful reader will learn from the operationalization protocol and the cohort design, and the test-set ambiguity is a textbook example of why reporting standards matter. But as written, the central claim is not yet supported.

My recommendation: send to peer review—this deserves a serious referee. Ask the authors to report all metrics separately for training, optimisation, and test, and to clarify where the 20 rater-2 scans sit in the split. If they do that, the tool could be a useful contribution. I wouldn't cite it in its current form.

Referee Report

3 major / 5 minor

Summary. The paper validates a fully automated deep-learning tool that predicts the Global Cortical Atrophy (GCA) score (0–39) from routine clinical CT brain scans of older patients. It uses 864 scans from an acute-medicine cohort, an acute-stroke cohort, and a legacy sample, with a stated 60/20/20 train/optimisation/test split. Visual GCA ratings by rater-1 on all 864 scans served as training labels; rater-2 rated a 20-scan subset. The reported agreement between the DL tool and rater-1 is MAE 3.2 and weighted kappa 0.45, with kappa 0.41 between tool and rater-2 and 0.28 between the two human raters. The abstract concludes that the tool 'measured GCA score accurately and without user input.'

Significance. Automated, fast (4 s/scan) GCA scoring on routine CT would be a genuinely useful tool for large-scale health-data research and potentially for clinical workflow, particularly because the GCA score is familiar to clinicians and the tool requires no user input. The study draws on real-world, multi-cohort CT data and provides a detailed operational protocol for the GCA scale, which is a useful contribution. However, the significance of the quantitative claims depends directly on whether the headline MAE/kappa values are computed on held-out test scans or include training/optimisation scans, and on whether a single rater's labels are a sufficiently reliable reference standard. The paper's own inter-rater data (rater-1 vs rater-2 kappa = 0.28 on 20 scans) make the latter concern concrete.

major comments (3)

[Results, first paragraph; Methods 'Prediction of GCA scores using a DL model'] The headline metrics are not tied to the 173-scan test set. Methods state a 60/20/20 split, but Results introduce the MAE with 'Among 864 patients ... MAE ... was 3.2 for the dataset overall', and the kappa values are not labelled by partition. Only the percentile error sentence ('For 88 (50%) of the testing set CT scans...') explicitly invokes the test set. If these MAE and kappa values include training/optimisation scans, they are in-sample fits to rater-1's labels and cannot support the generalisability claim. The authors must report test-set-only MAE and kappa, with subgroup values, and state explicitly whether the rater-2 subset of 20 was part of training/optimisation/test.
[Methods 'Application of the GCA scale'; Discussion, Limitations] The reference standard is a single rater: 'GCA scores from rater-1 were used as the ground truth during the DL-tool training', and all 864 validation labels come from that same rater. The paper's own inter-rater agreement, kappa = 0.28 (fair) and MAE = 5.2 between rater-1 and rater-2 on 20 scans, shows that the visual reference is not stable across raters. Agreement of the DL tool with rater-1 therefore partly measures how well the model learned that rater's idiosyncrasies, not an independent ground truth. The manuscript acknowledges this in the Limitations, but the abstract's 'accurately' claim is stronger than the evidence supports. Add intra-rater reliability for rater-1 and a larger multi-rater held-out validation.
[Results, Bland–Altman and statistical tests] The text uses non-significant paired t-tests and ANOVA (e.g., t = -0.43, p = 0.66; F = 1.06, p = 0.35) to support 'no difference' and thus agreement. Absence of a statistically significant difference is not evidence of equivalence, particularly in the n = 20 rater-2 subset. The agreement metrics (MAE, Bland–Altman limits, kappa) are more appropriate, but they should be reported for the test set and with confidence intervals; the p-values should not be the basis for claiming interchangeability.

minor comments (5)

[Abstract and Results] The abstract states 'Among 864 scans ... MAE ... was 3.2' without indicating that this is test-set-only. If the 3.2 MAE is in fact test-set-only, the text should say so explicitly; if not, it should be replaced with the test-set value.
[Results, classification accuracy] 'The accuracy of the DL-tool was 73% for mild atrophy and 70% for moderate and 70% for severe atrophy' is ambiguous. Is this per-class sensitivity, recall, or overall accuracy conditioned on true class? Please define and report cell counts, not just normalised percentages.
[Methods, Statistical Analysis] The repeated-measures one-way ANOVA comparing DL-tool, rater-1, and rater-2 is presumably on the 20 scans rated by rater-2, but the sample size is not stated in the Results. Please state n for each statistical comparison.
[Table 1 and Figure 1] Table 1 lists 'range=102-65 years' for ORCHARD-EPR; the order is reversed. Figure 1 caption refers to 'EC' but should identify rater-1 by the same label used elsewhere. Figure 6D legend has a typo: 'imapaired'.
[Supplementary Methods] The operationalisation protocol says 'the following scoring criteria has been developed' and provides a helpful manual, but the reliability results from the cited Hobden et al. study are not reproduced; since this protocol is central to the reference standard, a brief summary of those intra/inter-rater statistics would strengthen the paper.

Circularity Check

1 steps flagged · score 6.0 of 10

Headline accuracy metrics are reported on the full 864-scan dataset, not the held-out test set, so the central 'accurate' claim rests partly on in-sample fit to rater-1's training labels.

fitted input called prediction [Results, paragraph 2 (headline accuracy metrics)]
"The MAE for the difference between the DL-tool GCA scores and the GCA scores obtained by rater-1 was 3.2 for the dataset overall, 3.1 for ORCHARD-EPR, 3.3 for OCS, and 2.6 for the legacy scans (Figure 2A-D). ... Inter-rater agreement was Kappa=0.45 for the DL-tool vs rater-1, 0.41 for the DL-tool vs rater-2, and 0.28 for rater-1 vs rater-2."

Methods state that scans were divided 60/20/20 into training, optimisation, and testing sets, with rater-1 labels used as ground truth during training and tool testing performed on the testing set. The Results paragraph giving MAE=3.2 and kappa=0.45 describes 'the dataset overall' and never restricts these values to the 173-scan testing set; only later is the testing set invoked ('For 88 (50%) of the testing set CT scans...'). Because rater-1 labels for the 518 training and 173 optimisation scans were the fitting targets, MAE/kappa computed on the full 864 scans are in-sample performance measures. The abstract's central conclusion that the tool 'measured GCA score accurately' is supported by these unpartitioned statistics, so the primary accuracy claim reduces, at least in part, to the mod

full rationale

This is a validation study of a supervised DL model, not a first-principles derivation, so the learning itself is not circular: a held-out test set would provide genuine evidence of generalisation. However, the paper's headline accuracy statistics are not clearly restricted to the held-out test set. The methods describe a 60/20/20 split and testing on the testing set, but the Results report MAE and kappa 'for the dataset overall' (n=864), which includes training and optimisation scans. The only explicit testing-set result is the error distribution (50% within ±2, 75% within -5 to +4), not the headline MAE/kappa. Thus the central accuracy claim appears to rest substantially on in-sample agreement with rater-1's training labels. This is a concrete reduction: the model was fitted to rater-1 labels, and the reported error statistics are presented on the entire dataset without a test-set partition. The paper also acknowledges the single-rater training limitation, and the rater-1 vs rater-2 kappa of 0.28 is a separate validity concern, but that is not circularity. No other load-bearing self-citation or imported uniqueness argument was found; the self-citation to Hobden et al. in the supplementary methods supports the operationalisation but is not the central derivation. If the authors had clearly reported MAE/kappa only on the 173-scan test set, the circularity would be absent; as written, the primary quantitative claim is not demonstrably out-of-sample.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

The central claim rests on a fitted deep network trained on one rater's subjective scores. The free parameter is the undisclosed model itself. The main domain axioms are the validity of the single-rater ground truth and the assumption that CT morphology encodes the visual GCA scale. No new physical or conceptual entities are introduced.

free parameters (1)

DL model weights = not disclosed
The CNN regressor parameters are fit to 518 training scans labeled by rater-1; no architecture or weight values are provided, so the central prediction function is an undisclosed fitted object.

assumptions (4)

domain assumption Visual GCA ratings by a single trained rater are a valid and reliable reference standard for brain atrophy
Training and validation both use rater-1's scores; the paper reports only kappa 0.28 between rater-1 and rater-2 on 20 scans, so the reliability of this reference is weak.
domain assumption Brain atrophy is visually and measurably expressed in non-contrast CT morphology (sulcal and ventricular CSF spaces)
The whole method depends on CT intensity patterns representing atrophy; this is a standard radiological assumption, but the tool is trained on subjective ratings, not objective volumetry.
domain assumption The train/optimization/test random split produces independent test samples representative of the same scanner/cohort distribution
No external validation cohort is used; the random split from the same 864 scans means the test set shares scanner, protocol, and demographic characteristics with training.
standard math FSL-BET skull stripping and linear registration to a CT template preserve the atrophy-relevant information
Preprocessing is described as FSL-BET and linear registration (refs 12-14); failures of skull stripping on older, atrophic brains could directly distort the model input.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Validation of a CT-brain analysis tool for measuring global cortical atrophy in older patient cohorts." pith.science (2026). https://pith.science/paper/I2RGP5QH

@misc{pith2026250908012,
  author       = {Pith},
  title        = {Pith review of: Validation of a CT-brain analysis tool for measuring global cortical atrophy in older patient cohorts},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/I2RGP5QH}},
  note         = {Machine review of arXiv:2509.08012}
}

read the original abstract

Quantification of brain atrophy currently requires visual rating scales which are time consuming and automated brain image analysis is warranted. We validated our automated deep learning (DL) tool measuring the Global Cerebral Atrophy (GCA) score against trained human raters, and associations with age and cognitive impairment, in representative older (>65 years) patients. CT-brain scans were obtained from patients in acute medicine (ORCHARD-EPR), acute stroke (OCS studies) and a legacy sample. Scans were divided in a 60/20/20 ratio for training, optimisation and testing. CT-images were assessed by two trained raters (rater-1=864 scans, rater-2=20 scans). Agreement between DL tool-predicted GCA scores (range 0-39) and the visual ratings was evaluated using mean absolute error (MAE) and Cohen's weighted kappa. Among 864 scans (ORCHARD-EPR=578, OCS=200, legacy scans=86), MAE between the DL tool and rater-1 GCA scores was 3.2 overall, 3.1 for ORCHARD-EPR, 3.3 for OCS and 2.6 for the legacy scans and half had DL-predicted GCA error between -2 and 2. Inter-rater agreement was Kappa=0.45 between the DL-tool and rater-1, and 0.41 between the tool and rater- 2 whereas it was lower at 0.28 for rater-1 and rater-2. There was no difference in GCA scores from the DL-tool and the two raters (one-way ANOVA, p=0.35) or in mean GCA scores between the DL-tool and rater-1 (paired t-test, t=-0.43, p=0.66), the tool and rater-2 (t=1.35, p=0.18) or between rater-1 and rater-2 (t=0.99, p=0.32). DL-tool GCA scores correlated with age and cognitive scores (both p<0.001). Our DL CT-brain analysis tool measured GCA score accurately and without user input in real-world scans acquired from older patients. Our tool will enable extraction of standardised quantitative measures of atrophy at scale for use in health data research and will act as proof-of-concept towards a point-of-care clinically approved tool.

Figures

Figures reproduced from arXiv: 2509.08012 by the authors.

**Figure 2.** Scatter plots showing DL tool predicted GCA scores vs GCA scores from rater-1 showing a linear trend for (A) Overall dataset (B) ORCHARD-EPR (C) OCS (D) legacy scans (E) GCA scores from rater 1 vs GCA scores from rater 2 (F) GCA scores from rater 2 vs DL tool predicted GCA scores [PITH_FULL_IMAGE:figures/full_fig_p019_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

2 extracted references · 2 canonical work pages

[16]

Pendlebury ST, Klaus SP, Mather M, de Brito M, Wharton RM. Routine cognitive screening in older patients admitted to acute medicine: abbreviated mental test score (AMTS) and subjective memory complaint versus Montreal Cognitive Assessment and IQCODE. Age Ageing 2015;44(6):1000-1005. doi: 10.1093/ageing/afv134 17. Landis JR, Koch GG. The Measurement of Obs...

work page doi:10.1093/ageing/afv134 2015
[23]

MAGNIMS consensus recommendations on the use of brain and spinal cord atrophy measures in clinical practice

Sastre-Garriga J, Pareto D, Battaglini M, Rocca MA, Ciccarelli O, Enzinger C, Wuerfel J, Sormani MP, Barkhof F, Yousry TA, De Stefano N, Tintoré M, Filippi M, Gasperini C, Kappos L, Río J, Frederiksen J, Palace J, Vrenken H, Montalban X, Rovira À. MAGNIMS consensus recommendations on the use of brain and spinal cord atrophy measures in clinical practice. ...

work page doi:10.1038/s41582-020-0314-x 2020

Pith tools

Reviewed August 4, 2026 · model on record in the stance chip above.

[1] [16]

Pendlebury ST, Klaus SP, Mather M, de Brito M, Wharton RM. Routine cognitive screening in older patients admitted to acute medicine: abbreviated mental test score (AMTS) and subjective memory complaint versus Montreal Cognitive Assessment and IQCODE. Age Ageing 2015;44(6):1000-1005. doi: 10.1093/ageing/afv134 17. Landis JR, Koch GG. The Measurement of Obs...

work page doi:10.1093/ageing/afv134 2015

[2] [23]

MAGNIMS consensus recommendations on the use of brain and spinal cord atrophy measures in clinical practice

Sastre-Garriga J, Pareto D, Battaglini M, Rocca MA, Ciccarelli O, Enzinger C, Wuerfel J, Sormani MP, Barkhof F, Yousry TA, De Stefano N, Tintoré M, Filippi M, Gasperini C, Kappos L, Río J, Frederiksen J, Palace J, Vrenken H, Montalban X, Rovira À. MAGNIMS consensus recommendations on the use of brain and spinal cord atrophy measures in clinical practice. ...

work page doi:10.1038/s41582-020-0314-x 2020