Pith. sign in

REVIEW 3 major objections 5 minor 22 references

Xray2Xray: World Model from Chest X-rays with Volumetric Context

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Pith's one-line read A world model trained to predict how chest X-ray projections change across angles learns 3D chest structure, and its latents reach AUROC 0.809 for CVD risk prediction, outperforming supervised and self-supervised baselines.

desk verdict The paper's core idea is worth a look, but the headline CVD result currently rests on an unstated data-split assumption and an unfair baseline comparison. read the letter →

arxiv 2506.19055 v1 pith:IO6JGYOO submitted 2025-06-17 eess.IV cs.CV

classification eess.IVcs.CV
keywords worldmodelchestX-raylatentrepresentationtransitiondynamicstomographicreconstructioncardiovasculardiseaseriskself-supervisedpretrainingVQ-GAN
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Chest X-rays are 2D projections, so structures overlap and 3D context is lost. Xray2Xray is trained to predict how a projection changes as the imaging angle rotates, and the paper claims this next-projection objective forces the latent representation to encode the underlying chest volume. On cardiovascular disease risk estimation, the resulting 3D-aware latents reach AUROC 0.809, beating both supervised and self-supervised baselines, and they can be decoded into synthetic projections that support tomographic reconstruction. If the claim holds, 3D-style information could be recovered from routine frontal and lateral X-rays without an actual CT scan.

What carries the argument

The load-bearing mechanism is conditional next-projection prediction, written as $p(r_t \mid a_{t-1}, r_{t-1}, c_{t-1})$, where $r_t$ is the tokenized projection at angle $t$, $a_{t-1}$ is an action token encoding clockwise or counter-clockwise rotation, and $c_{t-1}$ is a class token summarizing the conditional state. A VQ-GAN encoder compresses each projection into a discrete token sequence; a GPT-style autoregressive Transformer predicts the next projection's token sequence; and during inference the model feeds its own output back in, rolling out a full angular trajectory. The class tokens from every step, taken from the middle Transformer layers, are concatenated and passed to an attention-based classifier for downstream tasks. The claim is that the only way to make consistent predictions across unseen angles is to internalize the volume, so the trajectory model's latent carries 3D information that a static 2D encoder cannot.

What would settle it

Check the NLST subject identifiers in the 1,000 pretraining CT volumes against the subject identifiers in the CVD risk train/validation/test splits. If any patient appears in both pretraining and downstream evaluation, the AUROC comparison is contaminated, and the claim that the world-model latents, rather than memorization, drive the gain is falsified. A cleaner test would retrain the model on publicly available CT volumes from a different cohort and rerun the CVD risk experiment on NLST.

Watch

Extended reading notes

Core claim

The central claim is that modeling the transition dynamics of X-ray projections—training a model to produce the tokenized next projection at a different angle given the current projection and a rotation-direction action—captures latent representations of the chest volume. The model combines a VQ-GAN vision encoder that turns each projection into a discrete token sequence with an autoregressive Transformer that predicts the next token sequence; it is pretrained on 37 projections spanning -90 to 90 degrees generated from 1,000 NLST CT volumes. At inference, an input frontal or lateral CXR is rolled forward along the trajectory, and the class tokens are aggregated by a classifier. The paper's reported empirical payoff is AUROC 0.809 for CVD risk estimation, above the best supervised baseline (BI-Mamba, 0.795) and well above iGPT (0.736) and CheXFound (0.757), plus CheXpert five-pathology AUROC 0.843 (within 2.8% of CheXFound's 0.871). It also reports that filtered back projection applied to synthesized projections yields recognizable tomographic slices, which it reads as evidence that the latent genuinely encodes 3D structure.

Load-bearing premise

The reported 0.809 AUROC for CVD risk assumes that the 1,000 NLST CT volumes used to pretrain Xray2Xray contain no patients who also appear in the 10,395 subjects used for CVD risk training, validation, or testing; the paper never states that the pretraining and downstream sets are disjoint at the patient level.

Editorial extensions

If this is right

  • A single frontal or lateral CXR can be expanded into a full angular sweep of synthetic projections spanning -90 to 90 degrees, and those projections can be reconstructed into tomographic slices with filtered back projection.
  • CVD risk estimation from two-view chest X-rays improves to AUROC 0.809, a gain of 1.4–7.3% over the supervised and self-supervised baselines reported in the paper.
  • The trajectory model transfers across datasets: after CycleGAN domain adaptation, it reaches AUROC 0.843 on five CheXpert pathologies, matching a supervised ConvNeXt and staying within 2.8% of a discriminative foundation model.
  • Because the representation is learned from CT-derived projections, its fidelity should scale with the number of CT volumes and the angular sampling density used in pretraining.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper does not state that the 1,000 NLST CT volumes used for pretraining are disjoint from the 10,395 NLST subjects used for CVD risk evaluation; if any subject overlaps, the reported AUROC gain could partly reflect leakage rather than 3D latents.
  • The same transition-dynamics objective could be applied to other projection imaging modalities, such as mammography, dental radiographs, or intraoperative fluoroscopy, where a rotating source or multiple views create a natural trajectory.
  • Synthesized projections from Xray2Xray could serve as data augmentation or as a regularization prior for sparse-view CT reconstruction, potentially reducing radiation dose in screening settings.
  • The 391M-parameter model includes a full trajectory rollout; a lighter downstream aggregator over class tokens could bring clinical deployment closer to real-time without retraining the world model.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes Xray2Xray, a generative 'world model' for chest X-rays composed of a VQ-GAN tokenizer and an autoregressive Transformer that predicts the token sequence of the next X-ray projection conditioned on the previous projection and a rotation action. The model is pretrained on simulated parallel-beam projections from 1,000 NLST CT volumes, spanning angles from -90 to +90 degrees in 5-degree steps. For downstream use, the authors freeze the pretrained model, infer class tokens along a trajectory from one or two input CXRs, and train a classifier on the concatenated frozen tokens. The method is evaluated for CVD risk prediction on NLST (AUROC 0.809), for five-pathology classification on CheXpert after CycleGAN domain transfer, and for projection synthesis and tomographic reconstruction quality via PSNR/SSIM. The paper's central claim is that modeling transition dynamics across X-ray projection angles yields latent representations that capture 3D volumetric context and improve downstream tasks.

Significance. If the results are reproducible and the leakage concerns are resolved, this is a valuable contribution to medical image representation learning. The idea of learning 3D volumetric context from 2D projections through a next-projection prediction objective is well motivated, and the two-stage discrete-autoencoder-plus-Transformer design is technically sound and clearly described. The paper gives a credit-worthy internal consistency check: synthesized projections are decoded and fed to filtered back-projection, showing that the latent trajectory retains enough angular information for tomographic reconstruction. The CVD evaluation includes several baselines and a DeLong statistical test, and the CheXpert evaluation tests generalization to a different dataset. The main limitations are experimental-control concerns in Section 3.1 that, if unaddressed, directly affect the strength of the headline claim.

major comments (3)
  1. [Section 3.1, Table 1] The manuscript does not state whether the 1,000 NLST CT volumes used for Xray2Xray pretraining are disjoint from the NLST subjects used for CVD risk estimation (7,268 training, 1,042 validation, 2,085 test). Both are drawn from the same NLST cohort and no exclusion step is described. If a subject appears in both sets, the pretraining objective has already observed CT-derived projections of that subject's anatomy, so the downstream latent representations can exploit memorized structure and inflate the reported AUROC of 0.809. This is especially consequential because the margin over BI-Mamba is not statistically significant (p=0.1326). Please report the exact patient-level overlap between the pretraining and downstream cohorts, and if overlap was not controlled, re-run the CVD experiment with fully disjoint patients and re-report all AUROC values and p-values.
  2. [Section 3.3, Tables 1 and 2] The advertised comparison is confounded by training data. Xray2Xray is pretrained on CT-derived simulated projections from NLST, while the ResNet-50, ViT-S, BI-Mamba, CheXFound, and iGPT baselines do not have access to this tomographic information (or to NLST pretraining in general). The reported gains therefore conflate the transition-modeling objective with the advantage of additional 3D training data. To support the claim that transition dynamics are responsible for the improvement, please include a controlled ablation: for example, a same-scale VQ-GAN/Transformer pretrained on the same CT projections with a non-transition objective (single-view reconstruction or shuffled-angle prediction), or report performance when the same pretraining data are made available to the comparison pipeline. Without such a control, the headline 'outperformed pretraining methods' statement is not established.
  3. [Section 3.4, Section 2.3] The claim that latent representations can reconstruct volumetric context from a real frontal or lateral CXR is not directly evaluated. Section 3.4 tests reconstruction from synthetic projections that Xray2Xray itself generates from an initial CT-derived projection; this is an internal consistency check and does not demonstrate that real CXRs (after CycleGAN domain transfer) contain enough information for the model to generate a useful CT volume. Section 2.3 states that 'we also verified the quality of the learned representation by reconstructing a CT volume from a frontal or lateral CXR,' but no such experiment or result is reported. Please add this experiment or soften the claim to 'synthetic X-ray projections' consistently in the Abstract and Section 2.3.
minor comments (5)
  1. [Section 2.2, Section 3.2] There are typos: 'trainsition' should be 'transition' in Section 2.2; 'Due the autoregressive process' should be 'Due to the autoregressive process'; and 'metircs' should be 'metrics' in Section 3.2.
  2. [Section 3.3, Table 3] Citations are inconsistent: Section 3.3 text cites CheXFound as [13], but Tables 1 and 2 and the reference list use [18]; Table 3 cites VQ-GAN as [20], while Section 2.1 uses [4]. Please harmonize citations.
  3. [Table 1] The Xray2Xray row uses '-' for its p-value, while p-values for other rows are compared against Xray2Xray. Please clarify the reference method for each p-value or list all pairwise comparisons explicitly.
  4. [Figure 2, Section 3.2] Figure 2 caption says 'Given an frontal-view projection' and should say 'Given a frontal-view projection'; it also uses 30-degree steps for illustration while Section 3.4 reports 5-degree steps, so please clarify the relationship between the qualitative and quantitative settings.
  5. [Section 3.2] The paper reports hyperparameters for VQ-GAN and the Transformer but does not specify CycleGAN training details or the exact procedure for domain transfer. Adding these details would improve reproducibility.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation found: Xray2Xray's next-projection prediction and downstream classifiers are trained on distinct objectives, and the self-citations appear only as non-load-bearing baselines.

full rationale

The paper's derivation chain is self-contained: VQ-GAN tokenizes projections, the Transformer maximizes log-likelihood of next-projection token sequences (Eq. 2), and downstream classifiers are trained on frozen class tokens (Eq. 3). The CVD AUROC and CheXpert results therefore do not reduce by construction to the pretraining objective; the synthesis/reconstruction evaluation compares model outputs to ground-truth projections and CT reconstructions, which is a genuine external check. The authors' own CheXFound, BI-Mamba, and GLoRI are used as baselines or classifier design components, but they are not invoked to justify the central claim that the latent representations encode 3D context, so the self-citations are not load-bearing. The possible overlap between the 1,000 NLST pretraining volumes and the NLST CVD evaluation subjects would be a data-leakage/correctness concern, not a circularity step under the stated criteria, because the paper does not define the CVD labels in terms of the pretraining objective.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The central claim rests on the tomographic principle that multi-angle projections determine the 3D volume (standard math), plus three domain assumptions: disjoint pretraining and evaluation splits, validity of CycleGAN domain transfer, and usefulness of middle-layer class tokens. No invented physical entities are introduced.

assumptions (4)
  • standard math Parallel-beam X-ray projections over [-90, 90] degrees are sufficient for tomographic reconstruction (Radon transform invertibility).
    Invoked in Section 3.1 to justify that 37 projections encode 3D spatial information.
  • domain assumption The 1,000 NLST CT volumes used for pretraining are disjoint from the subjects used in CVD risk evaluation.
    Section 3.1 describes pretraining on 1,000 NLST CT volumes and evaluation on 7,268+1,042+2,085 NLST subjects, but never states that these sets are disjoint. The main AUROC claim depends on this being true.
  • domain assumption CycleGAN-transformed real CXRs preserve the anatomical information needed for downstream diagnosis after matching the synthetic projection domain.
    Section 3.3 uses CycleGAN domain transfer before Xray2Xray feature extraction on CheXpert; no validation that the translation preserves pathology-relevant structure is provided.
  • domain assumption Middle-layer class tokens of the autoregressive transformer carry the most task-relevant spatial information.
    Section 2.3 states middle-layer features were used for classification, following [2,20]; no ablation for this model is shown.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Xray2Xray: World Model from Chest X-rays with Volumetric Context." pith.science (2026). https://pith.science/paper/IO6JGYOO

@misc{pith2026250619055,
  author       = {Pith},
  title        = {Pith review of: Xray2Xray: World Model from Chest X-rays with Volumetric Context},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/IO6JGYOO}},
  note         = {Machine review of arXiv:2506.19055}
}
read the original abstract

Chest X-rays (CXRs) are the most widely used medical imaging modality and play a pivotal role in diagnosing diseases. However, as 2D projection images, CXRs are limited by structural superposition, which constrains their effectiveness in precise disease diagnosis and risk prediction. To address the limitations of 2D CXRs, this study introduces Xray2Xray, a novel World Model that learns latent representations encoding 3D structural information from chest X-rays. Xray2Xray captures the latent representations of the chest volume by modeling the transition dynamics of X-ray projections across different angular positions with a vision model and a transition model. We employed the latent representations of Xray2Xray for downstream risk prediction and disease diagnosis tasks. Experimental results showed that Xray2Xray outperformed both supervised methods and self-supervised pretraining methods for cardiovascular disease risk estimation and achieved competitive performance in classifying five pathologies in CXRs. We also assessed the quality of Xray2Xray's latent representations through synthesis tasks and demonstrated that the latent representations can be used to reconstruct volumetric context.

Figures

Figures reproduced from arXiv: 2506.19055 by the authors.

Figure 1
Figure 1. Overall framework for Xray2Xray training. Xray2Xray is composed of a vision encoder and a transition model. We trained Xray2Xray with X-ray projections over different angular positions. The vision encoder first converted the input xk to a la￾tent token sequence rk. The transition model then predicted the next-projection rk+1 conditioned on rk. We used a cross-entropy loss to supervise Xray2Xray training. namics of p… view at source ↗
Figure 2
Figure 2. Xray2Xray synthesis results. Given an frontal-view projection (0◦ ) and actions as inputs, Xray2Xray synthesized the remaining projections across different angular positions without access to extra information. We used a step size of 30 degrees for illustration. and lateral radiographs in contrast to conventional methods that only encoded representations of 2D inputs without extrapolating 3D context. For disease dia… view at source ↗
Figure 3
Figure 3. Tomographic reconstruction with Xray2Xray’s synthetic projections. Top: Ground-truth reconstruction results at different positions along the z axis. Bottom: Reconstruction results with synthetic projections. We used the filtered back projection algorithm to reconstruct tomographic images from projections with a step size of 5 degrees. the VQ-GAN’s ability to compress an input into a latent token sequence by ex￾amini… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

22 extracted references · 9 canonical work pages

  1. [1]

    Nature Communica- tions12(1), 2963 (2021)

    Chao, H., Shan, H., Homayounieh, F., Singh, R., Khera, R.D., Guo, H., Su, T., Wang, G., Kalra, M.K., Yan, P.: Deep learning predicts cardiovascular disease risks from lung cancer screening low dose computed tomography. Nature Communica- tions12(1), 2963 (2021)

  2. [2]

    In: International conference on machine learning

    Chen, M., Radford, A., Child, R., Wu, J., Jun, H., Luan, D., Sutskever, I.: Gen- erative pretraining from pixels. In: International conference on machine learning. pp. 1691–1703. PMLR (2020)

  3. [3]

    In: International Conference on Learning Representations (2021)

    Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., Houlsby, N.: An image is worth 16x16 words: Transformers for image recognition at scale. In: International Conference on Learning Representations (2021)

  4. [4]

    In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition

    Esser, P., Rombach, R., Ommer, B.: Taming transformers for high-resolution image synthesis. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 12873–12883 (2021)

  5. [5]

    Academic radiology 17(12), 1560–1566 (2010)

    Feigin, D.S.: Lateral chest radiograph: a systematic approach. Academic radiology 17(12), 1560–1566 (2010)

  6. [6]

    arXiv preprint arXiv:1803.10122 (2018)

    Ha, D., Schmidhuber, J.: World models. arXiv preprint arXiv:1803.10122 (2018)

  7. [7]

    arXiv preprint arXiv:2301.04104 (2023)

    Hafner, D., Pasukonis, J., Ba, J., Lillicrap, T.: Mastering diverse domains through world models. arXiv preprint arXiv:2301.04104 (2023)

  8. [8]

    In: Proceedings of the IEEE conference on computer vision and pattern recognition

    He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 770–778 (2016)

Show all 22 references
  1. [9]

    In: Proceedings of the AAAI conference on artificial intelligence

    Irvin, J., Rajpurkar, P., Ko, M., Yu, Y., Ciurea-Ilcus, S., Chute, C., Marklund, H., Haghgoo, B., Ball, R., Shpanskaya, K., et al.: Chexpert: A large chest radiograph dataset with uncertainty labels and expert comparison. In: Proceedings of the AAAI conference on artificial in...

  2. [10]

    In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition

    Liu, Z., Mao, H., Wu, C.Y., Feichtenhofer, C., Darrell, T., Xie, S.: A convnet for the 2020s. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 11976–11986 (2022)

  3. [11]

    In: The Eleventh International Conference on Learning Representations (2023)

    Micheli, V., Alonso, E., Fleuret, F.: Transformers are sample-efficient world models. In: The Eleventh International Conference on Learning Representations (2023)

  4. [12]

    In: Developments in X-Ray Tomography XIV

    Niu, C., Dasegowda, G., Yan, P., Kalra, M.K., Wang, G.: Multi-view x-ray dis- sectography improves nodule detection. In: Developments in X-Ray Tomography XIV. vol. 12242, pp. 253–264. SPIE (2022)

  5. [13]

    arXiv preprint arXiv:2304.07193 (2023)

    Oquab, M., Darcet, T., Moutakanni, T., Vo, H., Szafraniec, M., Khalidov, V., Fernandez, P., Haziza, D., Massa, F., El-Nouby, A., et al.: Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193 (2023)

  6. [14]

    Cambridge university press (2017)

    Suetens, P.: Fundamentals of medical imaging. Cambridge university press (2017)

  7. [15]

    IEEE Signal Processing Letters21(11), 1389–1393 (2014) 10 Zefan Yang et al

    Sun, X., Xu, W.: Fast implementation of delong’s algorithm for comparing the areas under correlated receiver operating characteristic curves. IEEE Signal Processing Letters21(11), 1389–1393 (2014) 10 Zefan Yang et al

  8. [16]

    New England Journal of Medicine365(5), 395–409 (2011)

    Team, N.L.S.T.R.: Reduced lung-cancer mortality with low-dose computed tomo- graphic screening. New England Journal of Medicine365(5), 395–409 (2011)

  9. [17]

    In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R

    Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L.u., Polosukhin, I.: Attention is all you need. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processi...

  10. [18]

    arXiv preprint arXiv:2502.05142 (2025)

    Yang, Z., Xu, X., Zhang, J., Wang, G., Kalra, M.K., Yan, P.: Chest x-ray foun- dation model with global and local representations integration. arXiv preprint arXiv:2502.05142 (2025)

  11. [19]

    In: International Conference on Medical Image Computing and Computer-Assisted Intervention

    Yang, Z., Zhang, J., Wang, G., Kalra, M.K., Yan, P.: Cardiovascular disease de- tection from multi-view chest x-rays with bi-mamba. In: International Conference on Medical Image Computing and Computer-Assisted Intervention. pp. 134–144. Springer (2024)

  12. [20]

    arXiv preprint arXiv:2110.04627 (2021)

    Yu, J., Li, X., Koh, J.Y., Zhang, H., Pang, R., Qin, J., Ku, A., Xu, Y., Baldridge, J., Wu, Y.: Vector-quantized image modeling with improved vqgan. arXiv preprint arXiv:2110.04627 (2021)

  13. [21]

    arXiv preprint arXiv:2411.04983 (2024)

    Zhou, G., Pan, H., LeCun, Y., Pinto, L.: Dino-wm: World models on pre-trained visual features enable zero-shot planning. arXiv preprint arXiv:2411.04983 (2024)

  14. [22]

    In: Proceedings of the IEEE interna- tional conference on computer vision

    Zhu, J.Y., Park, T., Isola, P., Efros, A.A.: Unpaired image-to-image translation using cycle-consistent adversarial networks. In: Proceedings of the IEEE interna- tional conference on computer vision. pp. 2223–2232 (2017)

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.