REVIEW 3 major objections 5 minor 22 references
Xray2Xray: World Model from Chest X-rays with Volumetric Context
T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read A world model trained to predict how chest X-ray projections change across angles learns 3D chest structure, and its latents reach AUROC 0.809 for CVD risk prediction, outperforming supervised and self-supervised baselines.
desk verdict The paper's core idea is worth a look, but the headline CVD result currently rests on an unstated data-split assumption and an unfair baseline comparison. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is conditional next-projection prediction, written as $p(r_t \mid a_{t-1}, r_{t-1}, c_{t-1})$, where $r_t$ is the tokenized projection at angle $t$, $a_{t-1}$ is an action token encoding clockwise or counter-clockwise rotation, and $c_{t-1}$ is a class token summarizing the conditional state. A VQ-GAN encoder compresses each projection into a discrete token sequence; a GPT-style autoregressive Transformer predicts the next projection's token sequence; and during inference the model feeds its own output back in, rolling out a full angular trajectory. The class tokens from every step, taken from the middle Transformer layers, are concatenated and passed to an attention-based classifier for downstream tasks. The claim is that the only way to make consistent predictions across unseen angles is to internalize the volume, so the trajectory model's latent carries 3D information that a static 2D encoder cannot.
What would settle it
Check the NLST subject identifiers in the 1,000 pretraining CT volumes against the subject identifiers in the CVD risk train/validation/test splits. If any patient appears in both pretraining and downstream evaluation, the AUROC comparison is contaminated, and the claim that the world-model latents, rather than memorization, drive the gain is falsified. A cleaner test would retrain the model on publicly available CT volumes from a different cohort and rerun the CVD risk experiment on NLST.
Extended reading notes
Core claim
The central claim is that modeling the transition dynamics of X-ray projections—training a model to produce the tokenized next projection at a different angle given the current projection and a rotation-direction action—captures latent representations of the chest volume. The model combines a VQ-GAN vision encoder that turns each projection into a discrete token sequence with an autoregressive Transformer that predicts the next token sequence; it is pretrained on 37 projections spanning -90 to 90 degrees generated from 1,000 NLST CT volumes. At inference, an input frontal or lateral CXR is rolled forward along the trajectory, and the class tokens are aggregated by a classifier. The paper's reported empirical payoff is AUROC 0.809 for CVD risk estimation, above the best supervised baseline (BI-Mamba, 0.795) and well above iGPT (0.736) and CheXFound (0.757), plus CheXpert five-pathology AUROC 0.843 (within 2.8% of CheXFound's 0.871). It also reports that filtered back projection applied to synthesized projections yields recognizable tomographic slices, which it reads as evidence that the latent genuinely encodes 3D structure.
Load-bearing premise
The reported 0.809 AUROC for CVD risk assumes that the 1,000 NLST CT volumes used to pretrain Xray2Xray contain no patients who also appear in the 10,395 subjects used for CVD risk training, validation, or testing; the paper never states that the pretraining and downstream sets are disjoint at the patient level.
Editorial extensions
If this is right
- A single frontal or lateral CXR can be expanded into a full angular sweep of synthetic projections spanning -90 to 90 degrees, and those projections can be reconstructed into tomographic slices with filtered back projection.
- CVD risk estimation from two-view chest X-rays improves to AUROC 0.809, a gain of 1.4–7.3% over the supervised and self-supervised baselines reported in the paper.
- The trajectory model transfers across datasets: after CycleGAN domain adaptation, it reaches AUROC 0.843 on five CheXpert pathologies, matching a supervised ConvNeXt and staying within 2.8% of a discriminative foundation model.
- Because the representation is learned from CT-derived projections, its fidelity should scale with the number of CT volumes and the angular sampling density used in pretraining.
Reading between the lines
- The paper does not state that the 1,000 NLST CT volumes used for pretraining are disjoint from the 10,395 NLST subjects used for CVD risk evaluation; if any subject overlaps, the reported AUROC gain could partly reflect leakage rather than 3D latents.
- The same transition-dynamics objective could be applied to other projection imaging modalities, such as mammography, dental radiographs, or intraoperative fluoroscopy, where a rotating source or multiple views create a natural trajectory.
- Synthesized projections from Xray2Xray could serve as data augmentation or as a regularization prior for sparse-view CT reconstruction, potentially reducing radiation dose in screening settings.
- The 391M-parameter model includes a full trajectory rollout; a lighter downstream aggregator over class tokens could bring clinical deployment closer to real-time without retraining the world model.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes Xray2Xray, a generative 'world model' for chest X-rays composed of a VQ-GAN tokenizer and an autoregressive Transformer that predicts the token sequence of the next X-ray projection conditioned on the previous projection and a rotation action. The model is pretrained on simulated parallel-beam projections from 1,000 NLST CT volumes, spanning angles from -90 to +90 degrees in 5-degree steps. For downstream use, the authors freeze the pretrained model, infer class tokens along a trajectory from one or two input CXRs, and train a classifier on the concatenated frozen tokens. The method is evaluated for CVD risk prediction on NLST (AUROC 0.809), for five-pathology classification on CheXpert after CycleGAN domain transfer, and for projection synthesis and tomographic reconstruction quality via PSNR/SSIM. The paper's central claim is that modeling transition dynamics across X-ray projection angles yields latent representations that capture 3D volumetric context and improve downstream tasks.
Significance. If the results are reproducible and the leakage concerns are resolved, this is a valuable contribution to medical image representation learning. The idea of learning 3D volumetric context from 2D projections through a next-projection prediction objective is well motivated, and the two-stage discrete-autoencoder-plus-Transformer design is technically sound and clearly described. The paper gives a credit-worthy internal consistency check: synthesized projections are decoded and fed to filtered back-projection, showing that the latent trajectory retains enough angular information for tomographic reconstruction. The CVD evaluation includes several baselines and a DeLong statistical test, and the CheXpert evaluation tests generalization to a different dataset. The main limitations are experimental-control concerns in Section 3.1 that, if unaddressed, directly affect the strength of the headline claim.
major comments (3)
- [Section 3.1, Table 1] The manuscript does not state whether the 1,000 NLST CT volumes used for Xray2Xray pretraining are disjoint from the NLST subjects used for CVD risk estimation (7,268 training, 1,042 validation, 2,085 test). Both are drawn from the same NLST cohort and no exclusion step is described. If a subject appears in both sets, the pretraining objective has already observed CT-derived projections of that subject's anatomy, so the downstream latent representations can exploit memorized structure and inflate the reported AUROC of 0.809. This is especially consequential because the margin over BI-Mamba is not statistically significant (p=0.1326). Please report the exact patient-level overlap between the pretraining and downstream cohorts, and if overlap was not controlled, re-run the CVD experiment with fully disjoint patients and re-report all AUROC values and p-values.
- [Section 3.3, Tables 1 and 2] The advertised comparison is confounded by training data. Xray2Xray is pretrained on CT-derived simulated projections from NLST, while the ResNet-50, ViT-S, BI-Mamba, CheXFound, and iGPT baselines do not have access to this tomographic information (or to NLST pretraining in general). The reported gains therefore conflate the transition-modeling objective with the advantage of additional 3D training data. To support the claim that transition dynamics are responsible for the improvement, please include a controlled ablation: for example, a same-scale VQ-GAN/Transformer pretrained on the same CT projections with a non-transition objective (single-view reconstruction or shuffled-angle prediction), or report performance when the same pretraining data are made available to the comparison pipeline. Without such a control, the headline 'outperformed pretraining methods' statement is not established.
- [Section 3.4, Section 2.3] The claim that latent representations can reconstruct volumetric context from a real frontal or lateral CXR is not directly evaluated. Section 3.4 tests reconstruction from synthetic projections that Xray2Xray itself generates from an initial CT-derived projection; this is an internal consistency check and does not demonstrate that real CXRs (after CycleGAN domain transfer) contain enough information for the model to generate a useful CT volume. Section 2.3 states that 'we also verified the quality of the learned representation by reconstructing a CT volume from a frontal or lateral CXR,' but no such experiment or result is reported. Please add this experiment or soften the claim to 'synthetic X-ray projections' consistently in the Abstract and Section 2.3.
minor comments (5)
- [Section 2.2, Section 3.2] There are typos: 'trainsition' should be 'transition' in Section 2.2; 'Due the autoregressive process' should be 'Due to the autoregressive process'; and 'metircs' should be 'metrics' in Section 3.2.
- [Section 3.3, Table 3] Citations are inconsistent: Section 3.3 text cites CheXFound as [13], but Tables 1 and 2 and the reference list use [18]; Table 3 cites VQ-GAN as [20], while Section 2.1 uses [4]. Please harmonize citations.
- [Table 1] The Xray2Xray row uses '-' for its p-value, while p-values for other rows are compared against Xray2Xray. Please clarify the reference method for each p-value or list all pairwise comparisons explicitly.
- [Figure 2, Section 3.2] Figure 2 caption says 'Given an frontal-view projection' and should say 'Given a frontal-view projection'; it also uses 30-degree steps for illustration while Section 3.4 reports 5-degree steps, so please clarify the relationship between the qualitative and quantitative settings.
- [Section 3.2] The paper reports hyperparameters for VQ-GAN and the Transformer but does not specify CycleGAN training details or the exact procedure for domain transfer. Adding these details would improve reproducibility.
Circularity Check
No circular derivation found: Xray2Xray's next-projection prediction and downstream classifiers are trained on distinct objectives, and the self-citations appear only as non-load-bearing baselines.
full rationale
The paper's derivation chain is self-contained: VQ-GAN tokenizes projections, the Transformer maximizes log-likelihood of next-projection token sequences (Eq. 2), and downstream classifiers are trained on frozen class tokens (Eq. 3). The CVD AUROC and CheXpert results therefore do not reduce by construction to the pretraining objective; the synthesis/reconstruction evaluation compares model outputs to ground-truth projections and CT reconstructions, which is a genuine external check. The authors' own CheXFound, BI-Mamba, and GLoRI are used as baselines or classifier design components, but they are not invoked to justify the central claim that the latent representations encode 3D context, so the self-citations are not load-bearing. The possible overlap between the 1,000 NLST pretraining volumes and the NLST CVD evaluation subjects would be a data-leakage/correctness concern, not a circularity step under the stated criteria, because the paper does not define the CVD labels in terms of the pretraining objective.
Assumptions & free parameters
assumptions (4)
- standard math Parallel-beam X-ray projections over [-90, 90] degrees are sufficient for tomographic reconstruction (Radon transform invertibility).
- domain assumption The 1,000 NLST CT volumes used for pretraining are disjoint from the subjects used in CVD risk evaluation.
- domain assumption CycleGAN-transformed real CXRs preserve the anatomical information needed for downstream diagnosis after matching the synthetic projection domain.
- domain assumption Middle-layer class tokens of the autoregressive transformer carry the most task-relevant spatial information.
Cite this review
Pith. "Pith review of Xray2Xray: World Model from Chest X-rays with Volumetric Context." pith.science (2026). https://pith.science/paper/IO6JGYOO
@misc{pith2026250619055,
author = {Pith},
title = {Pith review of: Xray2Xray: World Model from Chest X-rays with Volumetric Context},
year = {2026},
howpublished = {\url{https://pith.science/paper/IO6JGYOO}},
note = {Machine review of arXiv:2506.19055}
}
read the original abstract
Chest X-rays (CXRs) are the most widely used medical imaging modality and play a pivotal role in diagnosing diseases. However, as 2D projection images, CXRs are limited by structural superposition, which constrains their effectiveness in precise disease diagnosis and risk prediction. To address the limitations of 2D CXRs, this study introduces Xray2Xray, a novel World Model that learns latent representations encoding 3D structural information from chest X-rays. Xray2Xray captures the latent representations of the chest volume by modeling the transition dynamics of X-ray projections across different angular positions with a vision model and a transition model. We employed the latent representations of Xray2Xray for downstream risk prediction and disease diagnosis tasks. Experimental results showed that Xray2Xray outperformed both supervised methods and self-supervised pretraining methods for cardiovascular disease risk estimation and achieved competitive performance in classifying five pathologies in CXRs. We also assessed the quality of Xray2Xray's latent representations through synthesis tasks and demonstrated that the latent representations can be used to reconstruct volumetric context.
Figures
Reference graph
Works this paper leans on
-
[1]
Nature Communica- tions12(1), 2963 (2021)
Chao, H., Shan, H., Homayounieh, F., Singh, R., Khera, R.D., Guo, H., Su, T., Wang, G., Kalra, M.K., Yan, P.: Deep learning predicts cardiovascular disease risks from lung cancer screening low dose computed tomography. Nature Communica- tions12(1), 2963 (2021)
work page 2021
-
[2]
In: International conference on machine learning
Chen, M., Radford, A., Child, R., Wu, J., Jun, H., Luan, D., Sutskever, I.: Gen- erative pretraining from pixels. In: International conference on machine learning. pp. 1691–1703. PMLR (2020)
2020
-
[3]
In: International Conference on Learning Representations (2021)
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., Houlsby, N.: An image is worth 16x16 words: Transformers for image recognition at scale. In: International Conference on Learning Representations (2021)
2021
-
[4]
In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition
Esser, P., Rombach, R., Ommer, B.: Taming transformers for high-resolution image synthesis. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 12873–12883 (2021)
2021
-
[5]
Academic radiology 17(12), 1560–1566 (2010)
Feigin, D.S.: Lateral chest radiograph: a systematic approach. Academic radiology 17(12), 1560–1566 (2010)
work page 2010
-
[6]
arXiv preprint arXiv:1803.10122 (2018)
Ha, D., Schmidhuber, J.: World models. arXiv preprint arXiv:1803.10122 (2018)
arXiv 2018
-
[7]
arXiv preprint arXiv:2301.04104 (2023)
Hafner, D., Pasukonis, J., Ba, J., Lillicrap, T.: Mastering diverse domains through world models. arXiv preprint arXiv:2301.04104 (2023)
arXiv 2023
-
[8]
In: Proceedings of the IEEE conference on computer vision and pattern recognition
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 770–778 (2016)
2016
Show all 22 references
-
[9]
In: Proceedings of the AAAI conference on artificial intelligence
Irvin, J., Rajpurkar, P., Ko, M., Yu, Y., Ciurea-Ilcus, S., Chute, C., Marklund, H., Haghgoo, B., Ball, R., Shpanskaya, K., et al.: Chexpert: A large chest radiograph dataset with uncertainty labels and expert comparison. In: Proceedings of the AAAI conference on artificial in...
2019
-
[10]
In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition
Liu, Z., Mao, H., Wu, C.Y., Feichtenhofer, C., Darrell, T., Xie, S.: A convnet for the 2020s. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 11976–11986 (2022)
2022
-
[11]
In: The Eleventh International Conference on Learning Representations (2023)
Micheli, V., Alonso, E., Fleuret, F.: Transformers are sample-efficient world models. In: The Eleventh International Conference on Learning Representations (2023)
2023
-
[12]
In: Developments in X-Ray Tomography XIV
Niu, C., Dasegowda, G., Yan, P., Kalra, M.K., Wang, G.: Multi-view x-ray dis- sectography improves nodule detection. In: Developments in X-Ray Tomography XIV. vol. 12242, pp. 253–264. SPIE (2022)
2022
-
[13]
arXiv preprint arXiv:2304.07193 (2023)
Oquab, M., Darcet, T., Moutakanni, T., Vo, H., Szafraniec, M., Khalidov, V., Fernandez, P., Haziza, D., Massa, F., El-Nouby, A., et al.: Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193 (2023)
2023 arXiv
-
[14]
Cambridge university press (2017)
Suetens, P.: Fundamentals of medical imaging. Cambridge university press (2017)
2017
-
[15]
IEEE Signal Processing Letters21(11), 1389–1393 (2014) 10 Zefan Yang et al
Sun, X., Xu, W.: Fast implementation of delong’s algorithm for comparing the areas under correlated receiver operating characteristic curves. IEEE Signal Processing Letters21(11), 1389–1393 (2014) 10 Zefan Yang et al
2014
-
[16]
New England Journal of Medicine365(5), 395–409 (2011)
Team, N.L.S.T.R.: Reduced lung-cancer mortality with low-dose computed tomo- graphic screening. New England Journal of Medicine365(5), 395–409 (2011)
2011
-
[17]
In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L.u., Polosukhin, I.: Attention is all you need. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processi...
2017
-
[18]
arXiv preprint arXiv:2502.05142 (2025)
Yang, Z., Xu, X., Zhang, J., Wang, G., Kalra, M.K., Yan, P.: Chest x-ray foun- dation model with global and local representations integration. arXiv preprint arXiv:2502.05142 (2025)
2025 arXiv
-
[19]
In: International Conference on Medical Image Computing and Computer-Assisted Intervention
Yang, Z., Zhang, J., Wang, G., Kalra, M.K., Yan, P.: Cardiovascular disease de- tection from multi-view chest x-rays with bi-mamba. In: International Conference on Medical Image Computing and Computer-Assisted Intervention. pp. 134–144. Springer (2024)
2024
-
[20]
arXiv preprint arXiv:2110.04627 (2021)
Yu, J., Li, X., Koh, J.Y., Zhang, H., Pang, R., Qin, J., Ku, A., Xu, Y., Baldridge, J., Wu, Y.: Vector-quantized image modeling with improved vqgan. arXiv preprint arXiv:2110.04627 (2021)
2021 arXiv
-
[21]
arXiv preprint arXiv:2411.04983 (2024)
Zhou, G., Pan, H., LeCun, Y., Pinto, L.: Dino-wm: World models on pre-trained visual features enable zero-shot planning. arXiv preprint arXiv:2411.04983 (2024)
2024 arXiv
-
[22]
In: Proceedings of the IEEE interna- tional conference on computer vision
Zhu, J.Y., Park, T., Isola, P., Efros, A.A.: Unpaired image-to-image translation using cycle-consistent adversarial networks. In: Proceedings of the IEEE interna- tional conference on computer vision. pp. 2223–2232 (2017)
2017
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.