REVIEW 4 major objections 5 minor 18 references
Enhancing Synthetic CT from CBCT via Multimodal Fusion and End-To-End Registration
T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read An end-to-end alignment step lifts multimodal synthetic CT quality in 79 of 90 test settings.
desk verdict A clean incremental study: adding a small affine STN to the authors' own multimodal sCT pipeline improves results under controlled synthetic misalignment, but the clinical generalization claim is not supported by the evidence. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing component is the Spatial Transformer Network (STN), a small convolutional network whose head outputs twelve parameters describing an affine transform (rotation, non-isotropic scaling, and translation). A grid sampler applies that transform to the preoperative CT, producing a resampled volume that is concatenated with the CBCT and fed into a 3D U-Net that synthesizes the CT. Because the STN sits inside the differentiable chain and is supervised by the reconstruction losses plus a registration loss, the alignment is learned for the synthesis task rather than as a separate preprocessing step.
What would settle it
Apply the same multimodal-plus-STN pipeline to a clinical dataset with measured non-rigid misalignment, such as inhale/exhale CT pairs, and compare against the perfectly aligned upper bound; if the STN gains over the fusion-only baseline vanish or invert once non-rigid displacement exceeds a few voxels, the central claim is limited to affine misalignment.
Extended reading notes
Core claim
The central claim is that explicitly registering the preoperative CT to the intraoperative CBCT inside the sCT generator, rather than feeding both volumes in as-is, makes the multimodal system produce better synthetic CTs. The registration is performed by a Spatial Transformer Network that estimates a single affine transform from the concatenated input volumes and resamples the CT accordingly; the whole pipeline is optimized end-to-end with a combination of voxel, structural-similarity, and perceptual losses plus a small registration loss. The controlled experiments show consistent gains over the multimodal baseline for moderate misalignment and degraded CBCT quality, while extreme misalignment reduces or reverses the benefit. The authors read this as evidence that learnable affine alignment captures clinically relevant anatomical discrepancies and is most valuable exactly where the intraoperative image is hardest to use.
Load-bearing premise
The controlled experiments assume that clinically relevant CBCT-CT misalignment is well represented by random affine warps (rotation, scaling, translation) controlled by a single strength parameter, whereas real patient misalignment also includes non-rigid deformation, breathing motion, and truncated fields of view; if that assumption fails, the benefit of a 12-parameter affine registration may not transfer to complex clinical settings.
Editorial extensions
If this is right
- Multimodal sCT pipelines can be trained end-to-end with internal alignment, removing the need for a separate pre-registration stage in many workflows.
- The main benefits appear precisely in the regimes where sCT is most needed: low-quality intraoperative CBCT combined with moderately misaligned preoperative CT.
- Perceptual and structural metrics can improve even in settings where voxelwise error does not, so evaluating sCT quality by a single pixelwise metric can miss the benefit of registration.
- At extreme misalignment, the affine STN loses its advantage, indicating that very large displacements still need an external initial alignment or a unimodal fallback.
- Because the method improves over the multimodal baseline in a large majority of settings across synthetic and clinical data, the affine registration component can serve as a default addition for typical CBCT-CT workflows.
Reading between the lines
- An extension the paper does not test is replacing the 12-parameter affine transform with a non-rigid or B-spline model; the reported failure at extreme misalignment suggests deformable motion is the next bottleneck.
- The results imply a deployment rule: measure expected misalignment first, and switch between multimodal-with-STN and unimodal sCT when misalignment exceeds roughly the αa=1 range used here.
- Further ablation could test whether the STN helps mainly by improving the CT signal fed into the U-Net or by regularizing the network's spatial assumptions; the registration-loss weight makes this separation testable.
- For clinical adoption, the method could be evaluated against a conventional external rigid registration followed by the same fusion network, to quantify how much end-to-end training contributes beyond pre-alignment alone.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes an end-to-end multimodal synthetic CT (sCT) pipeline that combines a 3D U-Net with a Spatial Transformer Network (STN) to register a preoperative CT to an intraoperative CBCT before early fusion and reconstruction. The STN predicts a 12-parameter affine transformation, and the model is trained with a weighted combination of MAE, 1-SSIM, and VGG-based perceptual losses. The authors evaluate on one synthetic dataset (CBCTLiTS) with controlled CBCT quality and affine misalignment, and on two clinical datasets (Pancreatic-CT-CBCT-SEG and SynthRAD2023). They report that the proposed MM+STN method outperforms the unimodal and multimodal baselines in 79/90 settings, with the largest gains for low CBCT quality and moderate misalignment. The paper also makes code publicly available.
Significance. If the central claim holds, the paper provides a useful incremental extension of prior multimodal sCT work [14] by explicitly modeling affine misalignment with a learnable registration module, supported by a controlled synthetic benchmark and public code with four random splits. The controlled study of CBCT quality and alignment is a strength, and the reproducibility measures (public code, standard deviations over splits) are commendable. However, the significance is limited by the narrow affine-only perturbation protocol used for the clinical datasets and by mixed results on SynthRAD, which are acknowledged but not fully reconciled with the abstract's global claim. The paper would be a reasonable contribution to a specialist venue, provided the generalizability claims are scaled to the evidence.
major comments (4)
- [Section 2.1 and Section 4] The claim of 'strong generalizability' to real-world clinical datasets is not supported because every dataset is unaligned with the same TorchIO RandomAffine procedure controlled by αa, and the STN predicts only 12 affine parameters. SynthRAD is explicitly rigidly registered before this synthetic perturbation, so neither clinical dataset provides an out-of-distribution test for non-rigid deformations, respiratory motion, or truncation artifacts, which are the clinically relevant forms of CBCT-CT misalignment. I recommend either adding an experiment with non-rigid perturbations (e.g. B-spline or demon-based deformation) or softening the generalizability conclusion to affine misalignment only.
- [Table 1 and Section 2.1] The pancreas dataset is described as including 'naturally occurring misalignment', yet Table 1 reports a CT-only baseline of 0.000±0.000 at αa=0 for all metrics, implying the CT input is exactly equal to the ground-truth CT. This is difficult to reconcile with natural misalignment and suggests that the preprocessing pipeline may have removed or ignored the real misalignment. Please clarify how the pancreatic dataset's native misalignment is handled and whether the reported αa=0 condition corresponds to perfectly aligned pairs.
- [Table 2 and Section 3] On the SynthRAD dataset, the proposed MM+STN method worsens MAE in four of the five αa settings (e.g. 0.120→0.127 at αa=1, 0.111→0.114 at αa=0.25) and worsens 1-SSIM in four of five settings (e.g. 0.136→0.142 at αa=1, 0.119→0.146 at αa=0.5). The text concedes the results are 'inconclusive', but these cases are still counted in the 79/90 aggregate improvements. Since sCT quality is primarily quantified by MAE and 1-SSIM, the abstract's claim that integrating registration improves sCT quality is overstated. Please report the 79/90 breakdown by dataset and metric and adjust the wording accordingly.
- [Tables 1-3] No significance testing or confidence intervals are reported. With only four random splits, many of the observed differences fall within one standard deviation of the baseline (e.g. pancreas αa=0.25 MAE: 0.105±0.038 vs 0.094±0.028; SynthRad αa=0.125 1-SSIM: 0.128±0.109 vs 0.130±0.114). To support the claim of improvement, the authors should provide paired significance tests (e.g. Wilcoxon signed-rank or paired t-test) or otherwise quantify the fraction of settings where the difference exceeds the noise level.
minor comments (5)
- [Section 2] The perceptual loss uses a 2D VGG16 network on 3D volumes, but the text does not specify how the 3D input is converted to 2D slices. Please clarify whether features are computed per axial slice, in all three orientations, or using some other scheme.
- [Introduction and Section 4] There are several typos, including 'preoperativce' in the Introduction, 'STM+MM' instead of 'STN+MM' in Section 3, and 'noticable' in Section 4. These should be corrected.
- [Figure 2] The figure caption and axis annotations are garbled (e.g. 'perfect alignment44120128'), making the visualization of the unalignment levels difficult to interpret. Please clean up the caption and axis labels.
- [Section 2.1] The sentence 'All datasets were unaligned in the same way' conflicts with the statement that the pancreas dataset has naturally occurring misalignment. Please describe the exact preprocessing for each dataset, including whether the natural misalignment is retained, corrected, or augmented.
- [Section 3] The sentence 'Overall, CT-only baselines show little difference between MM+STN and MM' is ambiguous. It should specify whether this refers to the CT-only column values or to the comparisons against the CT-only baseline.
Circularity Check
No significant circularity: the sCT improvement is an empirical, held-out comparison; self-citations are not load-bearing.
full rationale
Walk of the derivation chain: (1) The method trains an STN plus U-Net with a composite loss; the only direct supervision of the STN is the auxiliary registration term 10^-3*MSE(VCT, Y). This term supervises an intermediate alignment, not the reported sCT result. (2) The central claim—that adding the STN improves sCT quality in 79/90 settings—is established by comparing MAE/1-SSIM/perceptual metrics on held-out test splits under identical data splits and synthetic misalignments. The evaluation metric is the generated sCT versus the aligned CT; no reported number is defined as the training loss or as a fitted parameter. (3) The multimodal baseline [14] is the authors' predecessor, but it is reimplemented under the same conditions, so the comparison functions as an empirical ablation, not as an imported conclusion. (4) The CT-only baseline attains zero error at alpha_a=0 because its input equals its target; this is a sanity-check construction, and the paper does not present it as a predicted result. (5) The affine-only unalignment and application of the same synthetic unalignment to clinical datasets are external-validity limitations, not circular reductions. No uniqueness theorem or ansatz is smuggled in via self-citation; the STN is a standard component cited to Jaderberg et al. Overall, no load-bearing step reduces to its own inputs.
Assumptions & free parameters
free parameters (4)
- Loss weight alpha_1 (MAE) =
0.2
- Loss weight alpha_2 (1-SSIM) =
0.1
- Loss weight alpha_3 (perceptual) =
0.7
- Registration loss weight =
1e-3
assumptions (3)
- domain assumption Random affine unalignment with parameter alpha_a is a valid proxy for clinical CBCT-CT misalignment.
- domain assumption ImageNet-pretrained 2D VGG16 features are meaningful for 3D medical CT volumes.
- domain assumption Average metrics over four random splits without significance testing are sufficient to compare methods.
Cite this review
Pith. "Pith review of Enhancing Synthetic CT from CBCT via Multimodal Fusion and End-To-End Registration." pith.science (2026). https://pith.science/paper/DMRRRV6S
@misc{pith2026250706067,
author = {Pith},
title = {Pith review of: Enhancing Synthetic CT from CBCT via Multimodal Fusion and End-To-End Registration},
year = {2026},
howpublished = {\url{https://pith.science/paper/DMRRRV6S}},
note = {Machine review of arXiv:2507.06067}
}
read the original abstract
Cone-Beam Computed Tomography (CBCT) is widely used for intraoperative imaging due to its rapid acquisition and low radiation dose. However, CBCT images typically suffer from artifacts and lower visual quality compared to conventional Computed Tomography (CT). A promising solution is synthetic CT (sCT) generation, where CBCT volumes are translated into the CT domain. In this work, we enhance sCT generation through multimodal learning by jointly leveraging intraoperative CBCT and preoperative CT data. To overcome the inherent misalignment between modalities, we introduce an end-to-end learnable registration module within the sCT pipeline. This model is evaluated on a controlled synthetic dataset, allowing precise manipulation of data quality and alignment parameters. Further, we validate its robustness and generalizability on two real-world clinical datasets. Experimental results demonstrate that integrating registration in multimodal sCT generation improves sCT quality, outperforming baseline multimodal methods in 79 out of 90 evaluation settings. Notably, the improvement is most significant in cases where CBCT quality is low and the preoperative CT is moderately misaligned.
Figures
Reference graph
Works this paper leans on
-
[14]
Tschuchnig, M., Lamminger, L., Steininger, P., Gadermayr, M.: Enhancing syn- thetic ct from cbct via multimodal fusion: A study on the impact of cbct quality and alignment. arXiv preprint arXiv:2506.08716 (2025)
work page Pith review arXiv 2025
-
[1]
IEEE Transactions on Radiation and Plasma Medical Sciences (2025)
Altalib, A., McGregor, S., Li, C., Perelli, A.: Synthetic ct image generation from cbct: A systematic review. IEEE Transactions on Radiation and Plasma Medical Sciences (2025)
work page 2025
-
[2]
Medical Physics47(3), 1115–1125 (2020)
Chen, L., Liang, X., Shen, C., Jiang, S., Wang, J.: Synthetic ct generation from cbct images via deep learning. Medical Physics47(3), 1115–1125 (2020)
work page 2020
-
[3]
Physics in Medicine & Biology 66(11), 115019 (2021)
Chen, L., Liang, X., Shen, C., Nguyen, D., Jiang, S., Wang, J.: Synthetic ct gen- eration from cbct images via unsupervised deep learning. Physics in Medicine & Biology 66(11), 115019 (2021)
work page 2021
-
[4]
In: Medical Image Computing and Computer-Assisted Intervention
Çiçek, Ö., Abdulkadir, A., Lienkamp, S.S., Brox, T., Ronneberger, O.: 3d u-net: learning dense volumetric segmentation from sparse annotation. In: Medical Image Computing and Computer-Assisted Intervention. pp. 424–432 (2016)
work page 2016
-
[5]
In: IEEE Conference on Computer Vision and Pattern Recognition
Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: Imagenet: A large- scale hierarchical image database. In: IEEE Conference on Computer Vision and Pattern Recognition. pp. 248–255 (2009)
2009
-
[6]
Scientific Data 9(1), 637 (2022)
Hong, J., Reyngold, M., Crane, C., Cuaron, J., Hajj, C., Mann, J., Zinovoy, M., Yorke, E., LoCastro, E., Apte, A.P., et al.: Ct and cone-beam ct of ablative radi- ation therapy for pancreatic cancer with expert organ-at-risk contours. Scientific Data 9(1), 637 (2022)
work page 2022
-
[7]
Advances in Neural Information Processing Systems28 (2015)
Jaderberg, M., Simonyan, K., Zisserman, A., et al.: Spatial transformer networks. Advances in Neural Information Processing Systems28 (2015)
work page 2015
Show all 18 references
-
[8]
In: European Conference on Computer Vision–ECCV
Johnson, J., Alahi, A., Fei-Fei, L.: Perceptual losses for real-time style transfer and super-resolution. In: European Conference on Computer Vision–ECCV. pp. 694–711 (2016)
2016
-
[9]
Radiation Oncology 11, 1–10 (2016)
Oechsner, M., Chizzali, B., Devecka, M., Combs, S.E., Wilkens, J.J., Duma, M.N.: Registration uncertainties between 3d cone beam computed tomography and dif- ferent reference ct datasets in lung stereotactic body radiation therapy. Radiation Oncology 11, 1–10 (2016)
2016
-
[10]
In: Medical Image Comput- ing and Computer-Assisted Intervention
Podobnik, G., Strojan, P., Peterlin, P., Ibragimov, B., Vrtovec, T.: Multimodal ct and mr segmentation of head and neck organs-at-risk. In: Medical Image Comput- ing and Computer-Assisted Intervention. pp. 745–755 (2023)
2023
-
[11]
Otolaryngology—Head and Neck Surgery134(5), 801–808 (2006)
Rafferty,M.A.,Siewerdsen,J.H.,Chan,Y.,Daly,M.J.,Moseley,D.J.,Jaffray,D.A., Irish, J.C.: Intraoperative cone-beam ct for guidance of temporal bone surgery. Otolaryngology—Head and Neck Surgery134(5), 801–808 (2006)
2006
-
[12]
In: International Conference on Learning Representations (2015)
Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale im- age recognition. In: International Conference on Learning Representations (2015)
2015
-
[13]
Medical Physics50(7), 4664–4674 (2023)
Thummerer, A., van der Bijl, E., Galapon Jr, A., Verhoeff, J.J., Langendijk, J.A., Both, S., van den Berg, C.N.A., Maspero, M.: Synthrad2023 grand chal- lenge dataset: Generating synthetic ct for radiotherapy. Medical Physics50(7), 4664–4674 (2023)
2023
-
[15]
In: BVM Workshop
Tschuchnig, M.E., Coste-Marin, J., Steininger, P., Gadermayr, M.: Multi-task learning to improve semantic segmentation of cbct scans using image reconstruc- tion. In: BVM Workshop. pp. 243–248 (2024) Enhancing Synthetic CT via Multimodal Fusion and Registration 11
2024
-
[16]
In: Eurographics Work- shop on Visual Computing for Biology and Medicine (2024)
Tschuchnig, M.E., Steininger, P., Gadermayr, M.: CBCTLiTS: A Synthetic, Paired CBCT/CT Dataset For Segmentation And Style Transfer. In: Eurographics Work- shop on Visual Computing for Biology and Medicine (2024)
2024
-
[17]
Medical Physics (2024)
Wei, C., Albrecht, J., Rit, S., Laurendeau, M., Thummerer, A., Corradini, S., others, Landry, G.: Reduction of cone-beam ct artifacts in a robotic cbct device using saddle trajectories with integrated infrared tracking. Medical Physics (2024)
2024
-
[18]
In: IEEE Conference on Computer Vision and Pattern Recognition
Zhao, Z., Bai, H., Zhang, J., Zhang, Y., Zhang, K., Xu, S., Chen, D., Timofte, R., Van Gool, L.: Equivariant multi-modality image fusion. In: IEEE Conference on Computer Vision and Pattern Recognition. pp. 25912–25921 (2024) 12 M. Tschuchnig et al. T able 3. Evaluation results...
2024
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.