Pith. sign in

REVIEW 5 major objections 5 minor 1 cited by

Enhancing Synthetic CT from CBCT via Multimodal Fusion: A Study on the Impact of CBCT Quality and Alignment

T0 review · 5 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Adding a patient's planning CT to the intraoperative CBCT input yields synthetic CTs closer to true CT than CBCT-only translation, with the largest gains in well-aligned, low-quality CBCT cases.

desk verdict A useful 3D multimodal sCT study with a genuine contradiction in the Discussion; the central 'low-quality' claim rests on synthetic data, but the fusion idea mostly holds and the paper deserves review. read the letter →

arxiv 2506.08716 v1 pith:OUVNP3NM submitted 2025-06-10 eess.IV cs.CV

Maximilian Tschuchnig , Lukas Lamminger , Philipp Steininger , Michael Gadermayr This is my paper
classification eess.IVcs.CV
keywords syntheticCTcone-beamintraoperativeCBCTmultimodallearningearlyfusion3DU-NetCBCT-CTalignmentCBCTLiTS
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that synthetic CT generation from cone-beam CT should be treated as a multimodal problem: instead of translating the intraoperative CBCT to CT on its own, feed it together with the patient's preoperative planning CT and let a 3D U-Net fuse the two. The authors extend earlier two-dimensional multimodal synthetic-CT work to full 3D and use a controlled synthetic dataset in which CBCT quality and CBCT-CT misalignment are varied independently. They show the fused model beats both a CBCT-only and a CT-only baseline on MAE, SSIM, and perceptual similarity across quality and alignment settings, and the advantage is most pronounced when the CBCT is noisy and the volumes are well aligned. The same ordering largely reproduces on two real clinical datasets, with the caveats that perceptual similarity degrades under moderate-to-strong misalignment and SSIM suffers on a small, high-variability pancreas set. If the claim holds, a planning scan that is already available can clean up fast intraoperative imaging, which matters for dose reduction and real-time guidance.

What carries the argument

The working mechanism is early fusion inside a 3D U-Net: the CBCT and CT volumes are concatenated along a new channel axis, so the encoder's first convolution already sees both modalities and can combine them at every level of the feature hierarchy. The training objective is a weighted sum of mean absolute error, $1 - \mathrm{SSIM}$, and a perceptual loss computed on mid-level features of a pretrained deep image-classification network, with the perceptual term weighted highest ($\alpha_3 = 0.7$). The controlled experimental lever is the CBCTLiTS synthetic dataset, where CBCT quality is set by an undersampling parameter $\alpha_{\mathrm{np}}$ and misalignment by a single affine parameter $\alpha_a$; this separation is what lets the paper attribute performance differences to quality and alignment individually.

What would settle it

Measure the multimodal-versus-unimodal MAE gap on real CBCT-CT pairs with independently measured registration error and artifact level; the claim would fail if the gap disappears or reverses for clinically realistic misalignments around 6-10 mm or for artifact levels representative of low-dose acquisition. A simpler control on the same synthetic benchmark is to extend the $\alpha_a$ sweep beyond the tested maximum and check whether multimodal MAE stays below the unimodal baseline at large misalignment.

Watch

Extended reading notes

Core claim

The central claim is that early fusion of preoperative CT with intraoperative CBCT improves synthetic CT reconstruction over using either modality alone. A 3D U-Net whose input is the concatenated volume $[V_{\mathrm{CT}}, V_{\mathrm{CBCT}}]$ consistently beats the CBCT-only unimodal model on MAE, $1-\mathrm{SSIM}$, and perceptual loss across every CBCT quality level and every alignment level tested on the CBCTLiTS benchmark. It also beats the CT-only baseline in nearly all conditions, which the authors read as evidence that the network is genuinely combining the current anatomy in the CBCT with the high-quality structure of the CT rather than copying the planning scan. The advantage is largest for low-quality CBCT ($\alpha_{\mathrm{np}} = 32$) with good alignment, and it persists on the SynthRad pelvis dataset. On the pancreas dataset the multimodal model wins on MAE and perceptual loss at alignment $\alpha_a \le 0.25$ but loses on SSIM in all but the best-aligned cases.

Load-bearing premise

The load-bearing premise is that the synthetic CBCTLiTS volumes degrade like real clinical CBCT scans: the paper's ranking of gains is established on undersampling-based noise and random affine misalignment, and if those synthetic degradations do not reproduce real CBCT artifacts and patient motion, the central quality-and-alignment trend may not transfer to patients.

Editorial extensions

If this is right

  • With a registered planning CT available, multimodal early fusion reduces CBCT artifacts enough to improve synthetic CT across the tested quality and alignment range, so fusion is a direct upgrade over CBCT-only translation.
  • Because the largest gains occur at low CBCT quality, fusion is a robustness mechanism for fast or low-dose intraoperative acquisitions: degrading the CBCT does not erase the planning CT's contribution.
  • Performance improves steadily with alignment and shows no plateau, so better registration of CBCT to CT should convert directly into better synthetic CT, and alignment-aware architectures are a natural next step.
  • The CBCTLiTS synthetic benchmark predicts real-world behavior well enough, especially on SynthRad, that controlled synthetic studies can guide fusion design before clinical deployment.
  • Perceptual quality is the vulnerable point: under moderate-to-strong misalignment, fusion can lower MAE while slightly hurting high-frequency detail, so clinical use should pair fusion with adequate registration.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable extension the paper does not run: if fusion gains are largest when CBCT is poor, the same architecture could justify deliberately lowering intraoperative dose or projection count, with the planning CT compensating for the lost signal.
  • Because the model beats the CT-only baseline, it is not merely copying the planning scan, which raises the question of whether fusion can also track anatomical change between planning and treatment—something the perfectly aligned synthetic benchmark cannot assess.
  • The alignment sweep stops at an average displacement around 6.3 mm, while the paper cites clinical registration errors up to 10 mm; whether the multimodal advantage survives at larger misalignment is an open extrapolation.
  • The perceptual loss degradation under misalignment suggests that putting an explicit spatial-transformer or implicit registration module inside the U-Net, rather than relying on the network to register implicitly, is the most direct route to reclaiming the high-frequency detail that multimodal fusion currently loses.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper proposes a 3D U-Net with early fusion of intraoperative CBCT and preoperative CT to generate synthetic CT (sCT). It compares this multimodal model against CBCT-only and CT-only baselines on one synthetic dataset (CBCTLiTS) with controlled CBCT quality (alpha_np) and affine misalignment (alpha_a), and on two real datasets (Pancreatic-CT-CBCT-SEG and SynthRAD). The central claim is that multimodal sCT consistently outperforms unimodal baselines, with the largest gains for well-aligned, low-quality CBCT-CT pairs, and that these findings transfer to real clinical data.

Significance. The study addresses a clinically relevant problem and contributes a systematic, controlled analysis of alignment and quality for multimodal sCT, using public datasets and releasing code. If the ranking holds, early fusion of planning CT with intraoperative CBCT is a simple and effective way to improve CBCT-to-CT translation. The controlled synthetic setup is a useful benchmarking tool for studying alignment and quality factors. However, the paper's own results contain internal inconsistencies, and the real-world support is partial, so the significance depends on a revised, carefully qualified claim.

major comments (5)
  1. [Section 4 (Discussion) and Table 1] The statement that the multimodal model outperforms both CT-only and CBCT-only baselines 'across all alignment levels' is contradicted by Table 1: at alpha_a = 0 (perfect alignment) the CT-only baseline achieves MAE, 1-SSIM, and perceptual values of 0.000 for every alpha_np, while the multimodal model has nonzero values (e.g., for alpha_np = 32: 0.060 +/- 0.020, 0.086 +/- 0.044, and 0.187 +/- 0.019, respectively). Because CT-only is a unimodal baseline, the abstract's 'consistently outperform unimodal baselines' is too broad; the claim should be restricted to the CBCT-only comparison or to alpha_a > 0.
  2. [Section 5 (Conclusion) vs Abstract] The abstract states that the most significant gains are observed in 'well-aligned, low-quality CBCT-CT cases,' whereas the Conclusion states that multimodal sCT outperforms unimodal baselines 'especially under conditions of high CBCT quality and good alignment.' These are opposite claims. Please reconcile them and report the actual interaction with effect sizes (e.g., the gain at alpha_np = 32 versus alpha_np = 256 for each alpha_a).
  3. [Section 2.2 (Dataset) and Section 3 (Tables 2-3)] The low-quality interaction is derived exclusively from CBCTLiTS, where 'quality' is controlled by projection undersampling alpha_np and misalignment is affine. Real CBCT degradation includes scatter, beam hardening, and motion, and real misalignment is partly non-rigid. The real-world results do not control quality and only partially support the claim: on the pancreas dataset, improvements over CBCT-only are limited to alpha_a <= 0.25 for MAE and perceptual loss and alpha_a <= 0.125 for 1-SSIM, with SSIM degrading elsewhere; on SynthRad, perceptual gains appear only in well-aligned cases. Thus the abstract's assertion that the findings are 'highly reproducible in real-world clinical datasets' is not supported for the low-quality condition. Please qualify the claim or provide a real-world quality-controlled evaluation.
  4. [Section 2.1 (Loss) and Section 3 (Evaluation)] All three evaluation metrics (MAE, 1-SSIM, and perceptual loss) are terms in the training loss, with the same VGG-16 features used in the perceptual term. The comparison between models trained with the same objective is informative, but the reported absolute gains may partly reflect optimization of the training objective. Please add at least one metric that is not part of the training loss (e.g., PSNR or LPIPS with a different backbone) or a blinded expert rating.
  5. [Tables 1-3 and Section 3] The results are reported as means and standard deviations over four splits, but no significance testing is provided. Several differences described as improvements have overlapping or nearly overlapping standard deviations. Please report paired significance tests or confidence intervals for the multimodal-versus-unimodal comparisons, especially for the real datasets, to support the 'consistently outperform' claim.
minor comments (5)
  1. [Section 2.2] The alpha_a = 1 rotation range of up to +/- 22.5 degrees seems clinically extreme despite the reported mean voxel displacement of 7.8 voxels; please clarify whether the rotation and displacement ranges are consistent and whether such rotations are realistic in clinical practice.
  2. [Table 1] The table formatting is hard to parse: the header does not clearly label the CBCT-only unimodal column, and some entries appear merged in the typeset text. Please reformat and add clear column labels for all baselines.
  3. [Section 2.1] The statement that CT-only evaluation 'requires access to a true sCT ground truth, available only in the synthetic dataset' is confusing because the target in the real datasets is still CT; please clarify why CT-only is not applicable to the real datasets.
  4. [Figure 3] The six subplots are dense, and the dashed and dotted line styles combined with color encoding are difficult to distinguish in print; please enlarge the panels or separate the metric plots.
  5. [Authors and captions] There are minor typographical issues, including 'Steiningerr' in the author list and 'T able' in the table captions; please correct them.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the multimodal-vs-unimodal comparison is empirical and not forced by construction; metric/loss overlap is a measurement choice, and self-cited CBCTLiTS is a public dataset rather than a load-bearing theorem.

full rationale

The paper's central claim is an empirical comparison between models trained with the same objective. Although MAE, 1-SSIM, and the VGG perceptual loss appear in both the training loss and the evaluation metrics (Section 2.1 and Section 3), both multimodal and unimodal models are optimized and scored under the same functions, so the comparison is not a tautology and no prediction reduces to a fitted parameter. The CBCTLiTS dataset (Section 2.2) is self-cited, but it is a public, code-reproducible dataset with explicitly defined quality and alignment controls, and the quality/alignment trends are measured outcomes, not definitions of the conclusion. The claim that synthetic undersampling represents clinical CBCT artifacts is an external-validity assumption, not circularity. One internal inconsistency is noted for correctness rather than circularity: Table 1 shows CT-only reaching 0.000 error at perfect alignment (identity by construction), which contradicts Section 4's statement that multimodal outperforms CT-only across all alignment levels. This does not make the multimodal-vs-unimodal result circular.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

No new physical entities or theoretical constructs are introduced. The free parameters are standard training hyperparameters, with the loss weighting the most influential on the reported results. The assumptions are domain-level: synthetic data fidelity, transfer of ImageNet features, and representativeness of the evaluation splits.

free parameters (3)
  • Loss weights alpha_1, alpha_2, alpha_3 = 0.2, 0.1, 0.7
    Empirically chosen based on validation metrics and qualitative assessment (Section 2.1). All reported results depend on these weights, which favor perceptual loss.
  • U-Net feature map counts = 32, 64, 128, 256
    Architecture choice not justified by ablations; affects model capacity and memory (Section 2.1).
  • Volumetric downscale factor = 2
    Volumes isotropically downscaled by factor 2 due to the 48 GB VRAM limit (Section 2.3). This changes the effective resolution of all inputs and outputs.
assumptions (4)
  • domain assumption A 3D U-Net with early fusion can learn the CBCT-to-CT translation when trained with the weighted MAE, SSIM, and perceptual loss.
    The entire method rests on this trainability assumption; no formal guarantee is given (Section 2.1).
  • domain assumption CBCTLiTS synthetic CBCT, generated by simulating undersampling and affine misalignment, is a faithful proxy for clinical CBCT artifact and misalignment distributions.
    Used to derive the central alignment and quality findings (Section 2.2).
  • domain assumption ImageNet-pretrained VGG16 features, computed on 2D slices of 3D volumes, are a valid perceptual loss for CT image quality.
    The paper does not specify how 3D volumes are fed to VGG16 or why ImageNet features transfer to CT (Section 2.1).
  • domain assumption Four random 70/20/10 splits with identical misalignment draws give stable and representative evaluation.
    No statistical significance tests are reported; standard deviation across four runs is used instead (Section 2.3).

how reviews work

0 comments
Cite this review

Pith. "Pith review of Enhancing Synthetic CT from CBCT via Multimodal Fusion: A Study on the Impact of CBCT Quality and Alignment." pith.science (2026). https://pith.science/paper/OUVNP3NM

@misc{pith2026250608716,
  author       = {Pith},
  title        = {Pith review of: Enhancing Synthetic CT from CBCT via Multimodal Fusion: A Study on the Impact of CBCT Quality and Alignment},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/OUVNP3NM}},
  note         = {Machine review of arXiv:2506.08716}
}
read the original abstract

Cone-Beam Computed Tomography (CBCT) is widely used for real-time intraoperative imaging due to its low radiation dose and high acquisition speed. However, despite its high resolution, CBCT suffers from significant artifacts and thereby lower visual quality, compared to conventional Computed Tomography (CT). A recent approach to mitigate these artifacts is synthetic CT (sCT) generation, translating CBCT volumes into the CT domain. In this work, we enhance sCT generation through multimodal learning, integrating intraoperative CBCT with preoperative CT. Beyond validation on two real-world datasets, we use a versatile synthetic dataset, to analyze how CBCT-CT alignment and CBCT quality affect sCT quality. The results demonstrate that multimodal sCT consistently outperform unimodal baselines, with the most significant gains observed in well-aligned, low-quality CBCT-CT cases. Finally, we demonstrate that these findings are highly reproducible in real-world clinical datasets.

Figures

Figures reproduced from arXiv: 2506.08716 by the authors.

Figure 1
Figure 1. Proposed 3D, (early fusion) multimodal sCT model, based on U-Net. To train the model we use a weighted sum of voxel, patch based and per￾ceptual losses, as typically used in image reconstruction and image-to-image based models [8,2,1]. For a voxel based loss we apply mean absolute error (MAE) to focus on preserving image structure and robustness to noise. As a patch based loss, the structured similarity index measur… view at source ↗
Figure 2
Figure 2. Sample results, showing the original CT as well as synthetically unaligned versions of the same CT (CBCTLiTS dataset). Four different volumes are shown with increasing αa (unalignment), displaying rotation, scaling and minor translation. ment. It allows for controlled experiments to systematically study the effects of alignment and image quality on sCT generation. Nonetheless, real-world datasets remain essential fo… view at source ↗
Figure 3
Figure 3. Multimodal sCT results based on three metrics: MAE, 1-SSIM and perceptual dissimilarity. The x-axis indicates alignment level, from maximal unaligned to perfect alignment. The y-axis shows the mean value across four independent experiments. Solid lines represent our proposed multimodal method, with CBCT quality encoded by color. Dashed and dotted lines denote unimodal and CT-only baselines. CBCTLiTS results are show… view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Qualitative comparison of sCT reconstructions on the CBCTLiTS dataset. Rows correspond to different CBCT quality levels (top: low, bottom: high). Columns compare methods: a unimodal CBCT baseline (left) and multimodal reconstructions conditioned on preoperative CTs wit…

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Enhancing Synthetic CT from CBCT via Multimodal Fusion and End-To-End Registration

    eess.IV 2025-07 conditional novelty 5.0 of 10

    Adding an end-to-end affine registration module (a spatial transformer network) to multimodal synthetic CT generation improves output quality in 79 of 90 evaluated settings, especially with low-quality CBCT and modera...

Reference graph

Works this paper leans on

20 extracted references · 10 canonical work pages · cited by 1 Pith paper

  1. [1]

    IEEE Transactions on Radiation and Plasma Medical Sciences (2025)

    Altalib, A., McGregor, S., Li, C., Perelli, A.: Synthetic ct image generation from cbct: A systematic review. IEEE Transactions on Radiation and Plasma Medical Sciences (2025)

    2025

  2. [2]

    Medical Physics 47(3), 1115–1125 (2020)

    Chen, L., Liang, X., Shen, C., Jiang, S., Wang, J.: Synthetic ct generation from cbct images via deep learning. Medical Physics 47(3), 1115–1125 (2020)

    work page 2020

  3. [3]

    Physics in Medicine & Biology 66(11), 115019 (2021)

    Chen, L., Liang, X., Shen, C., Nguyen, D., Jiang, S., Wang, J.: Synthetic ct gen- eration from cbct images via unsupervised deep learning. Physics in Medicine & Biology 66(11), 115019 (2021)

    2021

  4. [4]

    In: Medical Image Computing and Computer-Assisted Intervention

    C ¸ i¸ cek,¨O., Abdulkadir, A., Lienkamp, S.S., Brox, T., Ronneberger, O.: 3d u-net: learning dense volumetric segmentation from sparse annotation. In: Medical Image Computing and Computer-Assisted Intervention. pp. 424–432 (2016)

    work page 2016

  5. [5]

    In: IEEE Conference on Computer Vision and Pattern Recognition

    Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: Imagenet: A large- scale hierarchical image database. In: IEEE Conference on Computer Vision and Pattern Recognition. pp. 248–255 (2009)

    2009

  6. [6]

    Scientific Data 9(1), 637 (2022)

    Hong, J., Reyngold, M., Crane, C., Cuaron, J., Hajj, C., Mann, J., Zinovoy, M., Yorke, E., LoCastro, E., Apte, A.P., et al.: Ct and cone-beam ct of ablative radi- ation therapy for pancreatic cancer with expert organ-at-risk contours. Scientific Data 9(1), 637 (2022)

    2022

  7. [7]

    Advances in Neural Information Processing Systems 28 (2015)

    Jaderberg, M., Simonyan, K., Zisserman, A., et al.: Spatial transformer networks. Advances in Neural Information Processing Systems 28 (2015)

    work page 2015

  8. [8]

    In: European Conference on Computer Vision–ECCV

    Johnson, J., Alahi, A., Fei-Fei, L.: Perceptual losses for real-time style transfer and super-resolution. In: European Conference on Computer Vision–ECCV. pp. 694–711 (2016)

    2016

Show all 20 references
  1. [9]

    Radiation Oncology 11, 1–10 (2016)

    Oechsner, M., Chizzali, B., Devecka, M., Combs, S.E., Wilkens, J.J., Duma, M.N.: Registration uncertainties between 3d cone beam computed tomography and dif- ferent reference ct datasets in lung stereotactic body radiation therapy. Radiation Oncology 11, 1–10 (2016)

    2016

  2. [10]

    In: Medical Image Comput- ing and Computer-Assisted Intervention

    Podobnik, G., Strojan, P., Peterlin, P., Ibragimov, B., Vrtovec, T.: Multimodal ct and mr segmentation of head and neck organs-at-risk. In: Medical Image Comput- ing and Computer-Assisted Intervention. pp. 745–755 (2023)

    2023

  3. [11]

    Otolaryngology—Head and Neck Surgery 134(5), 801–808 (2006) Enhancing Synthetic CT via Multimodal Fusion 13

    Rafferty, M.A., Siewerdsen, J.H., Chan, Y., Daly, M.J., Moseley, D.J., Jaffray, D.A., Irish, J.C.: Intraoperative cone-beam ct for guidance of temporal bone surgery. Otolaryngology—Head and Neck Surgery 134(5), 801–808 (2006) Enhancing Synthetic CT via Multimodal Fusion 13

    2006

  4. [12]

    In: International Conference on Learning Representations (2015)

    Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale im- age recognition. In: International Conference on Learning Representations (2015)

    2015

  5. [13]

    Medical Physics 50(7), 4664–4674 (2023)

    Thummerer, A., van der Bijl, E., Galapon Jr, A., Verhoeff, J.J., Langendijk, J.A., Both, S., van den Berg, C.N.A., Maspero, M.: Synthrad2023 grand chal- lenge dataset: Generating synthetic ct for radiotherapy. Medical Physics 50(7), 4664–4674 (2023)

    2023

  6. [14]

    In: BVM Workshop

    Tschuchnig, M.E., Coste-Marin, J., Steininger, P., Gadermayr, M.: Multi-task learning to improve semantic segmentation of cbct scans using image reconstruc- tion. In: BVM Workshop. pp. 243–248 (2024)

    2024

  7. [15]

    In: BVM Workshop

    Tschuchnig, M.E., Steininger, P., Gadermayr, M.: Improving segmentation by com- bining preoperative ct and intraoperative cbct using synthetic data. In: BVM Workshop. pp. 223–228 (2025)

    2025

  8. [16]

    In: Eurographics Work- shop on Visual Computing for Biology and Medicine (2024)

    Tschuchnig, M.E., Steininger, P., Gadermayr, M.: CBCTLiTS: A Synthetic, Paired CBCT/CT Dataset For Segmentation And Style Transfer. In: Eurographics Work- shop on Visual Computing for Biology and Medicine (2024)

    2024

  9. [17]

    Medical Physics (2024)

    Wei, C., Albrecht, J., Rit, S., Laurendeau, M., Thummerer, A., Corradini, S., others, Landry, G.: Reduction of cone-beam ct artifacts in a robotic cbct device using saddle trajectories with integrated infrared tracking. Medical Physics (2024)

    2024

  10. [18]

    In: Medical Image Computing and Computer Assisted Intervention

    Zhang, Y., Yang, J., Tian, J., Shi, Z., Zhong, C., Zhang, Y., He, Z.: Modality-aware mutual learning for multi-modal medical image segmentation. In: Medical Image Computing and Computer Assisted Intervention. pp. 589–599 (2021)

    2021

  11. [19]

    Image and Vision Computing105, 104042 (2021)

    Zhang, Y., Sidib´ e, D., Morel, O., M´ eriaudeau, F.: Deep multimodal fusion for semantic image segmentation: A survey. Image and Vision Computing105, 104042 (2021)

    2021

  12. [20]

    In: IEEE Conference on Computer Vision and Pattern Recognition

    Zhao, Z., Bai, H., Zhang, J., Zhang, Y., Zhang, K., Xu, S., Chen, D., Timofte, R., Van Gool, L.: Equivariant multi-modality image fusion. In: IEEE Conference on Computer Vision and Pattern Recognition. pp. 25912–25921 (2024)

    2024

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.