Pith. sign in

REVIEW 3 major objections 5 minor 1 cited by

Models Genesis: Generic Autodidactic Models for 3D Medical Image Analysis

T0 review · 3 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper reports that self-supervised 3D source models pretrained on unlabeled chest CTs, called Models Genesis, beat both training from scratch and 2D ImageNet-based transfer across five 3D medical imaging tasks.

desk verdict Worth refs: free 3D pre-trained models and a credible 3D-vs-2D transfer result, but the significance claims hinge on an unstated trial-resampling detail and the unified-framework attribution isn't supported by the ablation. read the letter →

arxiv 1908.06912 v1 pith:3MUETIR5 submitted 2019-08-19 eess.IV cs.CV

Zongwei Zhou , Vatsal Sodha , Md Mahfuzur Rahman Siddiquee , Ruibin Feng , Nima Tajbakhsh , Michael B. Gotway , Jianming Liang This is my paper · ORCID
classification eess.IVcs.CV
keywords self-supervisedlearning3DmedicalimageanalysistransferrestorationchestCTlungnodulesegmentationliverbraintumor
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that the standard transfer-learning recipe for medical imaging—pretraining on natural images and fine-tuning on 2D slices of 3D scans—throws away exactly the anatomical context that makes 3D imaging valuable. It proposes replacing the labeled source model with self-supervised source models pretrained directly on unlabeled chest CT volumes, and reports that these 'Models Genesis' outperform training from scratch in all five 3D target tasks and beat 2D ImageNet-based fine-tuning where the two are directly compared. A 2D version of the same recipe matches ImageNet fine-tuning within the same modality at zero annotation cost. If the reported results hold, large unlabeled medical archives can substitute for manually labeled pretraining data, and the 3D structure that slice-based transfer discards is learnable without labels.

What carries the argument

The load-bearing training scheme is image restoration: from an unlabeled image, patches $X$ of arbitrary size are cropped at random locations, transformed into $\tilde{X}$, and an encoder-decoder is trained to recover $X$ with an $L_1$ loss. Four transformations are consolidated under this single objective—a smooth monotonic intensity-remapping curve (appearance), local pixel shuffling inside small windows (texture), masking everything outside a complex window with a random value (global geometry), and filling the window with a constant (local continuity). Because one encoder-decoder handles all transformations, the representation is learned from multiple perspectives at once; the encoder initializes classification targets and the encoder-decoder initializes segmentation targets.

What would settle it

Rerun the five 3D target tasks with trial protocols that resample the training and validation splits each time (for example, repeated k-fold), and check whether Models Genesis still beats both scratch training and 2D ImageNet fine-tuning at p < 0.05; if the advantage disappears, the original significance claim depends on the unresolved protocol.

Watch

Extended reading notes

Core claim

The central claim is that a generic source model for 3D medical imaging can be created ex nihilo from unlabeled CT volumes, without any manual annotation, by training an encoder-decoder to restore original image patches from transformed versions. Fine-tuned on five target applications—lung nodule false-positive reduction, lung nodule segmentation, pulmonary embolism false-positive reduction, liver segmentation, and brain tumor segmentation—these models significantly outperform 3D training from scratch in every case, with p-values below 0.05. On the three tasks with a direct 2D comparison, the 3D Models Genesis also outperform fine-tuning from ImageNet and fine-tuning 2D versions of Models Genesis, while 3D scratch models do not consistently do so. The paper attributes the advantage to learning the recurrent anatomy of the chest from CT, arguing that anatomy itself provides supervision and that 3D context is the information 2D transfer paradigms sacrifice.

Load-bearing premise

The central significance claims rest on an unstated experimental detail: the paper says each result comes from 'at least 10 trials' but never says whether a trial resamples the training and validation data or only reinitializes the network on one fixed split; if only the random seed changes, the standard deviations capture seed noise and the reported p-values overstate the evidence.

Editorial extensions

If this is right

  • Models Genesis fine-tuning beats 3D training from scratch in every reported 3D task, with nominal p < 0.05 in all five cases.
  • Against 2D slice-based baselines, the 3D Models Genesis top both ImageNet fine-tuning and fine-tuning 2D versions of Models Genesis on all tasks compared, a claim that does not hold for 3D scratch models.
  • The 2D versions of Models Genesis offer performance equivalent to supervised ImageNet fine-tuning within the same modality, but with zero annotation cost.
  • The unified restoration objective is more stable across target tasks than any single transformation, and cross-domain transfer works across diseases, organs, datasets, and modalities, though same-domain transfer is preferred.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Inference: the same self-supervised recipe could be applied to MRI or ultrasound archives to build modality-specific source models; the paper lists this as future work, and its cross-modality restoration examples make it a concrete next test.
  • Inference: if the advantage is truly anatomical, the margin over 2D transfers should depend on the amount of 3D context available, so ablating slice count or input patch depth on the same tasks would be a direct check of the mechanism.
  • Inference: the pretrained encoder should be evaluated as a frozen feature extractor for anomaly detection or scan triage, uses the paper does not explore.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces Models Genesis, a set of self-supervised 3D models pre-trained on unlabeled chest CT images by restoring patches that have been altered with four transformations: non-linear intensity distortion, local pixel shuffling, out-painting, and in-painting. These transformations are unified into a single image-restoration task. The authors fine-tune the pre-trained encoder (and decoder) for downstream tasks and claim that, across five 3D applications, Models Genesis significantly outperform training from scratch, and also outperform 2D approaches including fine-tuning from ImageNet. The paper reports p-values for these gains, compares the unified scheme with individual schemes, provides a 2D version of the method, and releases the pre-trained models publicly.

Significance. If the reported results hold, the paper makes a useful contribution: it shows that self-supervised pre-training directly on 3D medical images can provide a reusable source model that transfers across diseases, organs, datasets, and even modalities, and it is one of the early systematic demonstrations that 3D self-supervised pre-training can beat 2D ImageNet transfer in 3D segmentation and classification tasks. The authors are careful to pre-train on a disjoint subset of LIDC-IDRI so that held-out LIDC-IDRI tests are not contaminated by the proxy-task training data, and they make their pre-trained models publicly available. Reproducibility is also aided by the use of standard architectures (3D U-Net and ResNet-18-based U-Net) and public benchmarks.

major comments (3)
  1. [Sec. 3, Experiment protocol] The protocol states that performance was measured 'through at least 10 trials' and analyzed with an 'independent two-sample t-test', but it never defines what a trial is. If a trial re-initializes the network on one fixed train/validation/test split, then the trials are not independent draws from the data distribution, the standard deviations in Tables 2 and 4 reflect only seed/optimization noise, and the reported p-values overstate dataset-level significance. This is load-bearing for the smallest and most variable gains in Table 2: BMS (90.16±0.41 vs. 90.60±0.20, p=0.0041) and NCC (94.25±5.07 vs. 98.20±0.51, p=0.0180). The authors must state whether the patient-level split is resampled per trial, report the exact number of trials per condition, and either supply a split-resampling analysis or appropriately qualify the significance claims.
  2. [Table 3 and Appendix H, Fig. 15] Table 3 reports that the unified framework is not significantly better than the best individual scheme in any of the five tasks (p=0.0848, 0.0520, 0.2102, 0.4249, 0.4276), yet the abstract and Section 4 attribute the performance of Models Genesis to 'our unified self-supervised learning framework' and claim that the unified scheme is more robust. Appendix H, Fig. 15, caption even states 'p<0.05' for this comparison, directly contradicting Table 3. Either the statistical analysis is wrong in one of these places, or the claim that the unified framework is the driver of the gains needs to be substantially softened; the current text overstates what the experiments show.
  3. [Sec. 4 Conclusion and Table 2 footnotes] The conclusion states that the empirical results are 'surpassing state-of-the-art performances in most of the applications', but the footnotes of Table 2 compare against published results using different metrics and protocols. For LCS, the LiTS winner's 96.60% is a Dice score with postprocessing, while the reported 91.13% is an IoU without postprocessing; for BMS, the BraTS winner's 91.00% uses ensembling while the reported 92.58% does not. Dice and IoU are not interchangeable, so these footnotes do not support a state-of-the-art claim. The authors should either present metric-matched, protocol-matched comparisons or remove the 'surpassing state-of-the-art' wording.
minor comments (5)
  1. [Sec. 3, Experiment protocol] The exact number of trials is never stated; 'at least 10' is not sufficient to verify the t-test degrees of freedom or to assess whether the standard deviations are stable. Please report the exact n per condition and describe how seeds and data splits are handled.
  2. [Sec. 1 and Sec. 3] The introduction says Models Genesis 'significantly outperform learning from scratch in all seven target tasks', but Table 2 provides significance tests for only five 3D tasks, and Fig. 2 does not report significance for the 2D tasks. Please align the text with the actual statistical analyses.
  3. [Table 4] In Table 4 the 2D Genesis columns are numerically below the 2D ImageNet columns for all three tasks (e.g., NCC: 97.45 vs. 97.79), yet the abstract's 'consistently top any 2D approaches' could be read as referring to all Models Genesis variants. The wording should make explicit that the superiority claim is for the 3D Genesis model.
  4. [Appendix H, Fig. 15 caption] The caption states 'yielding either the best result or comparable performance to the best model (p<0.05)', which is inconsistent with the p-values in Table 3; this needs correction regardless of which analysis is intended.
  5. [Throughout] There are several typos and formatting issues: 'Genesis Check CT' in the Table 4 caption should be 'Genesis Chest CT', and 'Appendix4' appears without a space. A careful proofread would improve readability.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: target performance is measured on held-out external benchmarks after fine-tuning a self-supervised source model; no fitted parameter is renamed as a prediction.

full rationale

The paper's claimed derivation chain is an empirical transfer-learning protocol, not a derivation of a target quantity from an input. Models Genesis are pre-trained on unlabeled CT/X-ray via self-supervised restoration (Sec. 2 and Appendix B–E), and target AUC/IoU is obtained by fine-tuning on held-out public benchmarks (LUNA2016, LIDC-IDRI, LiTS2017, BraTS2013, ChestX-ray8). The protocol explicitly separates pre-training and testing data: 'The reason that we decided not to use all images in LIDC-IDRI and in ChestX-ray8 for training Models Genesis is to avoid test-image leaks between proxy and target tasks, so that we can confidently use the rest images solely for testing Models Genesis as well as the target models.' Thus the headline numbers are not written into the model by construction. The significance statistics rest on an under-specified trial design ('through at least 10 trials' with an 'independent two-sample t-test'), and the caption for Fig. 2 describes Genesis Chest CT 2D as trained on 'LUNA dataset' without stating a disjoint split for the NCC target; these are reproducibility/statistical-reporting gaps, not circularity, because no metric is an algebraic function of a fitted input. Self-citations to Tajbakhsh et al. [9,10] are contextual and not load-bearing, and no uniqueness theorem or ansatz is imported from the authors' prior work. I find no step where a prediction reduces to its input by definition.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The contribution is empirical, so the ledger records the unstated experimental choices the claims depend on. The transformation probabilities, window sizes, masked-region cap, pre-training scan count, and shuffle-window count are hand-chosen constants whose values are partly unreported. The four axioms are the domain priors and statistical assumptions that carry the significance claims; the trial-independence axiom is the most fragile. No new physical or model-level entities are postulated; Models Genesis are trained neural networks and the four transformations are data-augmentation methods, not invented entities with independent falsifiable handles.

free parameters (5)
  • predefined transformation probabilities = not reported
    Fig. 4 states each transformation is applied with pre-defined probabilities; the actual values are never given, and they control the mix of self-supervised tasks in the unified framework.
  • local shuffling window sizes = 2D: 5x5 to 20x20; 3D: unreported
    Appendix Sec. C shows window size controls task difficulty and is kept below the receptive field; the exact sampling distribution for 3D patches is not specified.
  • masked-region cap = under 1/4 of patch
    Appendix Secs. D-E set an ad hoc cap on how much of a patch may be out-painted or in-painted to keep the restoration task solvable.
  • pre-training dataset size = 534 CT scans from LIDC-IDRI
    Chosen to avoid test-image leakage between proxy and target tasks; how the 534 were selected from LIDC-IDRI is not specified.
  • local shuffling window count = 1000 windows per patch
    Appendix Sec. C: 'randomly select 1,000 windows from the patch', a hand-chosen constant.
assumptions (4)
  • domain assumption Pixel intensities in CT (Hounsfield units) and their relative order in other modalities carry anatomical meaning.
    Motivates the non-linear intensity transformation (Sec. 2, Sec. B); standard radiology knowledge, load-bearing only for that scheme.
  • domain assumption Learning to restore transformed patches produces a representation that transfers to classification and segmentation.
    The central hypothesis of the paper; tested empirically, not derived.
  • ad hoc to paper The ten training trials are independent replications.
    Underlies every t-test p-value in Tables 2-4; the paper does not say whether trials resample splits or only seeds.
  • domain assumption The held-out test images are disjoint from the 534 pre-training scans.
    Required for the leakage-avoidance protocol in Sec. 3; the exact split is not auditable from the text.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Models Genesis: Generic Autodidactic Models for 3D Medical Image Analysis." pith.science (2026). https://pith.science/paper/3MUETIR5

@misc{pith2026190806912,
  author       = {Pith},
  title        = {Pith review of: Models Genesis: Generic Autodidactic Models for 3D Medical Image Analysis},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/3MUETIR5}},
  note         = {Machine review of arXiv:1908.06912}
}
read the original abstract

Transfer learning from natural image to medical image has established as one of the most practical paradigms in deep learning for medical image analysis. However, to fit this paradigm, 3D imaging tasks in the most prominent imaging modalities (e.g., CT and MRI) have to be reformulated and solved in 2D, losing rich 3D anatomical information and inevitably compromising the performance. To overcome this limitation, we have built a set of models, called Generic Autodidactic Models, nicknamed Models Genesis, because they are created ex nihilo (with no manual labeling), self-taught (learned by self-supervision), and generic (served as source models for generating application-specific target models). Our extensive experiments demonstrate that our Models Genesis significantly outperform learning from scratch in all five target 3D applications covering both segmentation and classification. More importantly, learning a model from scratch simply in 3D may not necessarily yield performance better than transfer learning from ImageNet in 2D, but our Models Genesis consistently top any 2D approaches including fine-tuning the models pre-trained from ImageNet as well as fine-tuning the 2D versions of our Models Genesis, confirming the importance of 3D anatomical information and significance of our Models Genesis for 3D medical imaging. This performance is attributed to our unified self-supervised learning framework, built on a simple yet powerful observation: the sophisticated yet recurrent anatomy in medical images can serve as strong supervision signals for deep models to learn common anatomical representation automatically via self-supervision. As open science, all pre-trained Models Genesis are available at https://github.com/MrGiovanni/ModelsGenesis.

Figures

Figures reproduced from arXiv: 1908.06912 by the authors.

Figure 1
Figure 1. Our unified self-supervised learning framework consolidates four novel transfor [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Comparison of 2D solutions on four 2D target tasks. To investigate the same- and [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 3
Figure 3. Overview of our unified self-supervised learning framework. Given an image, [PITH_FULL_IMAGE:figures/full_fig_p010_3.png] view at source ↗
Figures from the paper (13 more)
Figure 4
Figure 4. Figure 4: [Better viewed on-line in color and zoomed in for details] Our novel unified self [PITH_FULL_IMAGE:figures/full_fig_p011_4.png]
Figure 5
Figure 5. Figure 5: We adopt non-linear intensity transformation as a new training scheme for self [PITH_FULL_IMAGE:figures/full_fig_p013_5.png]
Figure 6
Figure 6. Figure 6: We adopt local pixel shuffling as a new training scheme for self-supervised learn [PITH_FULL_IMAGE:figures/full_fig_p015_6.png]
Figure 7
Figure 7. Figure 7: We adopt out-painting as a new training scheme for self-supervised learning, [PITH_FULL_IMAGE:figures/full_fig_p017_7.png]
Figure 8
Figure 8. Figure 8: We adopt in-painting as a new training scheme for self-supervised learning, [PITH_FULL_IMAGE:figures/full_fig_p018_8.png]
Figure 9
Figure 9. Figure 9: Qualitative assessment of CT image restoration quality using Models Genesis [PITH_FULL_IMAGE:figures/full_fig_p019_9.png]
Figure 10
Figure 10. Figure 10: Continued from Fig [PITH_FULL_IMAGE:figures/full_fig_p020_10.png]
Figure 11
Figure 11. Figure 11: Qualitative assessment of image restoration quality by Genesis Chest CT across [PITH_FULL_IMAGE:figures/full_fig_p021_11.png]
Figure 12
Figure 12. Figure 12: Qualitative comparisons of Genesis Chest X-ray trained with unified self [PITH_FULL_IMAGE:figures/full_fig_p022_12.png]
Figure 13
Figure 13. Figure 13: Continued from Fig [PITH_FULL_IMAGE:figures/full_fig_p023_13.png]
Figure 14
Figure 14. Figure 14: Qualitative results of image restoration from Genesis Chest X-ray across [PITH_FULL_IMAGE:figures/full_fig_p024_14.png]
Figure 15
Figure 15. Figure 15: Comparison of Models Genesis and Models ImageNet. In the top three sub [PITH_FULL_IMAGE:figures/full_fig_p025_15.png]
Figure 16
Figure 16. Figure 16: Fine-tuning the pre-trained NiftyNet vs. training it from scratch. The results [PITH_FULL_IMAGE:figures/full_fig_p026_16.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Brain Imaging Foundation Models, Are We There Yet? A Systematic Review of Foundation Models for Brain Imaging and Biomedical Research

    eess.IV 2025-06 conditional novelty 6.0 of 10

    A systematic review of brain imaging foundation models covering 86 models and 161 datasets, with a performance tournament, dataset atlas, and duplicated-data warnings.

Reference graph

Works this paper leans on

18 extracted references · 17 canonical work pages · cited by 1 Pith paper

  1. [1]

    : The lung image database consortium (lidc) and image database resource initiative (idri): a completed reference database of lung nodules on ct scans

    Armato III, S.G., et al. : The lung image database consortium (lidc) and image database resource initiative (idri): a completed reference database of lung nodules on ct scans. Medical physics, 38(2), 915–931 (2011) 19, 21, 24

    work page 2011

  2. [2]

    In: Springer Handbook of Medical Technol- ogy, 311–342 (2011) 13

    Buzug, T.M.: Computed tomography. In: Springer Handbook of Medical Technol- ogy, 311–342 (2011) 13

    work page 2011

  3. [3]

    : Self-Supervised GANs via Auxiliary Rotation Loss

    Chen, T., et al. : Self-Supervised GANs via Auxiliary Rotation Loss. In: CVPR, 12154-12163 (2019) 12

    work page 2019

  4. [4]

    : ImageNet: A large-scale hierarchical image database

    Deng, J., et al. : ImageNet: A large-scale hierarchical image database. In: CVPR, 248–255 (2009) 12

    work page 2009

  5. [5]

    Springer Science & Business Media (2012) 13

    Forbes, G.B.: Human body composition: growth, aging, nutrition, and activity. Springer Science & Business Media (2012) 13

    work page 2012

  6. [6]

    TMI, 37(8), 1822–1834 (2018) 26

    Gibson, E., et al.: Automatic multi-organ segmentation on abdominal ct with dense v-networks. TMI, 37(8), 1822–1834 (2018) 26

    work page 2018

  7. [7]

    : Niftynet: a deep-learning platform for medical imag- ing

    Gibson, E., et al. : Niftynet: a deep-learning platform for medical imag- ing. Computer Methods and Programs in Biomedicine (2018), https://www. sciencedirect.com/science/article/pii/S0169260717311823 26

    work page 2018

  8. [8]

    Hurst, R.T., et al. : Incidence of subclinical atherosclerosis as a marker of car- diovascular risk in retired professional football players.The American journal of cardiology, 105(8), 1107–1111 (2010) 21, 24

    work page 2010

Show all 18 references
  1. [9]

    ACM Transac- tions on Graphics (ToG), 36(4), 107 (2017) 16

    Iizuka, S., et al.: Globally and locally consistent image completion. ACM Transac- tions on Graphics (ToG), 36(4), 107 (2017) 16

    2017

  2. [10]

    : Self-supervised visual feature learning with deep neural networks: A survey

    Jing, L., et al. : Self-supervised visual feature learning with deep neural networks: A survey. arXiv:1902.06162 (2019) 12

    2019 arXiv

  3. [11]

    : Imagenet classification with deep convolutional neural net- works

    Krizhevsky, A., et al. : Imagenet classification with deep convolutional neural net- works. In: NIPS, 1097–1105 (2012) 12

    2012

  4. [12]

    : Revisiting self-supervised visual representation learning

    Kolesnikov, A., et al. : Revisiting self-supervised visual representation learning. In: CVPR, 1920–1929 (2019) 12

    2019

  5. [13]

    : The multimodal brain tumor image segmentation benchmark (brats)

    Menze, B.H., et al. : The multimodal brain tumor image segmentation benchmark (brats). TMI, 34(10), 1993 (2015) 21, 24

    2015

  6. [14]

    Industrial Press Inc

    Mortenson, M.E.: Mathematics for computer graphics applications. Industrial Press Inc. (1999) 13

    1999

  7. [15]

    : Context encoders: Feature learning by inpainting

    Pathak, D., et al. : Context encoders: Feature learning by inpainting. In: CVPR, 2536–2544 (2016) 16

    2016

  8. [16]

    Journal of machine learning research 11(Dec), 3371–3408 (2010) 16

    Vincent, P., et al.: Stacked denoising autoencoders: Learning useful representations in a deep network with a local denoising criterion. Journal of machine learning research 11(Dec), 3371–3408 (2010) 16

    2010

  9. [17]

    In: CVPR, 2097–2106 (2017) 21, 22, 24

    Wang, X., et al.: Chestx-ray8: Hospital-scale chest x-ray database and benchmarks on weakly-supervised classification and localization of common thorax diseases. In: CVPR, 2097–2106 (2017) 21, 22, 24

    2017

  10. [18]

    Journal of digital imaging, 32(2), 290-299 (2019)

    Zhou, Z., et al.: Integrating active learning and transfer learning for carotid intima- media thickness video interpretation. Journal of digital imaging, 32(2), 290-299 (2019). 21, 24 27

    2019

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.