REVIEW 4 major objections 2 minor 1 cited by
This systematic review of 46 publicly available abdominal CT datasets finds 59.1% case reuse and 75.3% geographic skew toward North America and Europe, with domain shift and selection bias prevalent in larger datasets, undermining AI genera
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
A review of 46 public abdominal CT datasets finds substantial case overlap and geographic skew, threatening the real-world applicability of AI models.
T0 review reviewed 2026-08-05 challenge →
load-bearing objection A potentially valuable quantitative review of abdominal CT dataset bias, but the headline percentages are not assessable from the abstract alone. the 4 major comments →
State of Abdominal CT Datasets: A Critical Review of Bias, Clinical Relevance, and Real-world Applicability
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
Core claim
The paper reports a systematic review of 46 publicly available abdominal CT datasets totaling 50,256 studies, claiming that these datasets are substantially redundant (59.1% of cases are reused across datasets) and geographically skewed (75.3% originate from North America and Europe). For the 19 datasets with at least 100 cases, the most common high-risk biases are domain shift (63%) and selection bias (57%), both of which threaten the generalizability of AI models trained on them, especially to resource-limited healthcare environments.
What carries the argument
The central mechanism is a systematic-review protocol applied to the dataset itself: a structured search to identify 46 public abdominal CT datasets, a redundancy check to quantify case reuse across datasets, a geographic-origin classification, and a bias-risk assessment restricted to the 19 datasets with at least 100 cases. These steps produce the headline percentages and the identification of domain shift and selection bias as the dominant high-risk categories.
Load-bearing premise
The set of 46 datasets this review selected is a complete and unbiased representation of all publicly available abdominal CT datasets; if the search missed or misclassified any substantial number of datasets, the reported percentages would change.
What would settle it
Independently re-run the dataset search with a broader set of inclusion criteria and compute the overlap of imaging studies across all publicly available abdominal CT datasets using image-level similarity matching; if the true case-reuse rate falls well below 59.1% and the geographic balance shifts materially away from a Western majority, the central percentages of this review would not hold.
If this is right
- AI models trained on existing public abdominal CT datasets may perform poorly in hospitals outside North America and Europe, especially in resource-limited settings.
- Benchmark results reported on these datasets likely overstate real-world performance because the same patients' images appear across multiple training and test sets.
- Dataset developers should prioritize multi-institutional data collection covering diverse populations, scanner manufacturers, and imaging protocols rather than adding more cases from the same sources.
- Reporting standards for public medical datasets should include explicit statements about patient overlap and geographic composition so downstream users can interpret model evaluations correctly.
Where Pith is reading between the lines
- The same redundancy and geographic-skew patterns likely affect public datasets for other anatomical regions and imaging modalities, though this review only quantifies abdominal CT.
- A testable extension would be to measure per-dataset patient-level diversity (age, sex, body habitus, comorbid conditions) and correlate it with the reported bias categories, since the review does not provide such patient-level breakdowns.
- The review's emphasis on resource-limited settings suggests a concrete remedy: curating datasets that oversample underrepresented populations and older scanner technology, then benchmarking model degradation across these subgroups.
- If the 59.1% case-reuse figure holds, it implies that many published AI performance numbers for abdominal CT are partly memorization of a small pool of unique patients, which would inflate confidence in those systems when deployed to new institutions.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This systematic review claims to examine 46 publicly available abdominal CT datasets (50,256 studies) and reports two headline quantitative findings: a 59.1% case-reuse rate across datasets and a 75.3% geographic skew toward North America and Europe. For the 19 datasets with at least 100 cases, the authors report high-risk bias categories of domain shift (63%) and selection bias (57%). The abstract also proposes dataset improvement strategies, including multi-institutional collaboration, standardized protocols, and inclusion of diverse populations. Because the full text was not available for review, the assessment is limited to the abstract, which contains no details on search strategy, inclusion criteria, coding definitions, or statistical methods.
Significance. If the reported figures are accurate, this review addresses an important and under-reported problem in medical imaging AI: redundancy and geographic bias in public abdominal CT datasets directly threaten model generalizability and equity. The topic is timely, and the quantitative claims—59.1% case reuse and 75.3% Western skew—would be valuable evidence for dataset developers and downstream users. The paper also offers constructive recommendations. However, the significance is currently conditional on the existence of a rigorous, reproducible methodology; the abstract alone provides no evidence of such methodology, so the contribution cannot yet be assessed as a scientific result. I credit the authors for undertaking a systematic review with explicit numerical outcomes, but the lack of verifiable methods in the available text means the claims are currently unsubstantiated.
major comments (4)
- [Abstract, first paragraph] The abstract reports concrete figures (46 datasets, 50,256 studies, 59.1% case reuse, 75.3% geographic skew) but provides no search protocol, inclusion/exclusion criteria, or method for identifying eligible datasets. The reader's weakest assumption—that the 46 datasets form a complete and unbiased representation—is load-bearing. Without a reproducible search strategy and a PRISMA-style flow diagram, the percentages could shift materially if even a few large non-Western datasets were missed or misclassified. This is a verification gap that must be closed before the central claims can be accepted.
- [Abstract, 'case reuse' definition] The 59.1% case-reuse rate depends entirely on the operational definition of 'reuse.' Ambiguities include whether reuse means the same patient appearing in multiple datasets, duplicate image series, or derived/cropped versions of the same underlying study. The abstract does not define the unit of analysis (patient, study, series, image) or the threshold for considering two cases 'reused.' Without a codebook and inter-rater reliability assessment, the redundancy claim is not independently checkable.
- [Abstract, second paragraph (bias assessment)] The bias assessment is limited to 19 datasets with >=100 cases, but the abstract does not define the assessment framework for 'domain shift' or 'selection bias.' Were these categories scored by a standardized tool (e.g., PROBAST or QUADAS-2), by expert judgment, or by quantitative metrics? The 63% and 57% rates are meaningless without a clear rubric and evidence that the assessments were reproducible. This is a central methodological component and must be specified.
- [Abstract, geographic skew] The 75.3% 'from North America and Europe' claim requires a definition of geographic attribution: institution country, funding source, patient origin, or dataset publication venue? The term 'Western/geographic skew' conflates geography with socioeconomic classification. If attribution is based on institution country, the number will differ from patient-origin-based attribution. The abstract provides no such definition, so the claim's precision is unclear.
minor comments (2)
- [Abstract, general] The abstract does not mention a protocol registration or adherence to systematic review guidelines such as PRISMA. Including this information would improve transparency. Also, the phrase 'high-risk categories' could be clarified as 'high risk of bias' to avoid ambiguity about what is at risk.
- [Abstract, terminology] The term 'Western/geographic skew' is informal. Suggest using specific geographic regions (e.g., 'North America and Europe') or a clearly defined socioeconomic classification, and avoid conflating 'Western' with a geographic measure.
Circularity Check
No circularity: the review's percentages are empirical counts over datasets, not derived quantities defined in terms of themselves.
full rationale
This is a systematic review of publicly available abdominal CT datasets. The central claims—59.1% case reuse and 75.3% Western/geographic skew—are presented as empirical aggregates from inspection of 46 datasets (50,256 studies). There is no derivation chain, no fitted parameter, and no equation in which an output is defined in terms of the same output. The paper does not appear to rely on self-citation as load-bearing evidence, and the 'bias assessment' is described as a categorical evaluation of dataset characteristics, not a model that predicts its own inputs. The abstract-only availability limits verification of the search protocol, inclusion criteria, and coding rules, but those are transparency and reproducibility concerns, not circularity. A systematic review that counts dataset properties cannot be circular simply because the numbers depend on what datasets were included; that dependency is ordinary empirical measurement, not a self-definitional reduction. Therefore, no specific circular step can be quoted, and the appropriate score is 0.
Axiom & Free-Parameter Ledger
Cite this review
Pith. "Pith review of State of Abdominal CT Datasets: A Critical Review of Bias, Clinical Relevance, and Real-world Applicability." pith.science (2026). https://pith.science/paper/S5YLGLU6
@misc{pith2026250813626,
author = {Pith},
title = {Pith review of: State of Abdominal CT Datasets: A Critical Review of Bias, Clinical Relevance, and Real-world Applicability},
year = {2026},
howpublished = {\url{https://pith.science/paper/S5YLGLU6}},
note = {Machine review of arXiv:2508.13626}
}
read the original abstract
This systematic review critically evaluates publicly available abdominal CT datasets and their suitability for artificial intelligence (AI) applications in clinical settings. We examined 46 publicly available abdominal CT datasets (50,256 studies). Across all 46 datasets, we found substantial redundancy (59.1\% case reuse) and a Western/geographic skew (75.3\% from North America and Europe). A bias assessment was performed on the 19 datasets with >=100 cases; within this subset, the most prevalent high-risk categories were domain shift (63\%) and selection bias (57\%), both of which may undermine model generalizability across diverse healthcare environments -- particularly in resource-limited settings. To address these challenges, we propose targeted strategies for dataset improvement, including multi-institutional collaboration, adoption of standardized protocols, and deliberate inclusion of diverse patient populations and imaging technologies. These efforts are crucial in supporting the development of more equitable and clinically robust AI models for abdominal imaging.
Forward citations
Cited by 1 Pith paper
-
Do Medical Foundation Models Generalize on the African Brain?
Medical foundation models show no consistent generalization gap on African brain MRI; performance differences track dataset size, not data origin.
This paper was first reviewed by deepseek-v4-flash on August 5, 2026.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.