Pith. sign in

REVIEW 4 major objections 5 minor 18 references

Enhancing Synthetic CT from CBCT via Multimodal Fusion and End-To-End Registration

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Pith's one-line read An end-to-end alignment step lifts multimodal synthetic CT quality in 79 of 90 test settings.

desk verdict A clean incremental study: adding a small affine STN to the authors' own multimodal sCT pipeline improves results under controlled synthetic misalignment, but the clinical generalization claim is not supported by the evidence. read the letter →

arxiv 2507.06067 v1 pith:DMRRRV6S submitted 2025-07-08 eess.IV cs.AIcs.CV

classification eess.IVcs.AIcs.CV
keywords syntheticCTCBCTmultimodalfusionimageregistrationspatialtransformernetworkdeeplearningmedicalimaging
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that a multimodal synthetic CT (sCT) pipeline that fuses intraoperative CBCT with preoperative CT should include an explicit, trainable alignment step rather than relying on the fusion network to cope with misalignment on its own. The authors add a Spatial Transformer Network that predicts a 12-parameter affine transform and resamples the CT onto the CBCT grid, trained jointly with the reconstruction U-Net. Across a controlled synthetic dataset and two clinical datasets, the added registration improves sCT quality over the fusion-only baseline in 79 of 90 evaluated settings, with the largest gains when CBCT quality is low and the CT is moderately misaligned. The paper concludes that learnable affine registration is a practical component for multimodal sCT generation, especially in degraded intraoperative imaging conditions.

What carries the argument

The load-bearing component is the Spatial Transformer Network (STN), a small convolutional network whose head outputs twelve parameters describing an affine transform (rotation, non-isotropic scaling, and translation). A grid sampler applies that transform to the preoperative CT, producing a resampled volume that is concatenated with the CBCT and fed into a 3D U-Net that synthesizes the CT. Because the STN sits inside the differentiable chain and is supervised by the reconstruction losses plus a registration loss, the alignment is learned for the synthesis task rather than as a separate preprocessing step.

What would settle it

Apply the same multimodal-plus-STN pipeline to a clinical dataset with measured non-rigid misalignment, such as inhale/exhale CT pairs, and compare against the perfectly aligned upper bound; if the STN gains over the fusion-only baseline vanish or invert once non-rigid displacement exceeds a few voxels, the central claim is limited to affine misalignment.

Watch

Extended reading notes

Core claim

The central claim is that explicitly registering the preoperative CT to the intraoperative CBCT inside the sCT generator, rather than feeding both volumes in as-is, makes the multimodal system produce better synthetic CTs. The registration is performed by a Spatial Transformer Network that estimates a single affine transform from the concatenated input volumes and resamples the CT accordingly; the whole pipeline is optimized end-to-end with a combination of voxel, structural-similarity, and perceptual losses plus a small registration loss. The controlled experiments show consistent gains over the multimodal baseline for moderate misalignment and degraded CBCT quality, while extreme misalignment reduces or reverses the benefit. The authors read this as evidence that learnable affine alignment captures clinically relevant anatomical discrepancies and is most valuable exactly where the intraoperative image is hardest to use.

Load-bearing premise

The controlled experiments assume that clinically relevant CBCT-CT misalignment is well represented by random affine warps (rotation, scaling, translation) controlled by a single strength parameter, whereas real patient misalignment also includes non-rigid deformation, breathing motion, and truncated fields of view; if that assumption fails, the benefit of a 12-parameter affine registration may not transfer to complex clinical settings.

Editorial extensions

If this is right

  • Multimodal sCT pipelines can be trained end-to-end with internal alignment, removing the need for a separate pre-registration stage in many workflows.
  • The main benefits appear precisely in the regimes where sCT is most needed: low-quality intraoperative CBCT combined with moderately misaligned preoperative CT.
  • Perceptual and structural metrics can improve even in settings where voxelwise error does not, so evaluating sCT quality by a single pixelwise metric can miss the benefit of registration.
  • At extreme misalignment, the affine STN loses its advantage, indicating that very large displacements still need an external initial alignment or a unimodal fallback.
  • Because the method improves over the multimodal baseline in a large majority of settings across synthetic and clinical data, the affine registration component can serve as a default addition for typical CBCT-CT workflows.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An extension the paper does not test is replacing the 12-parameter affine transform with a non-rigid or B-spline model; the reported failure at extreme misalignment suggests deformable motion is the next bottleneck.
  • The results imply a deployment rule: measure expected misalignment first, and switch between multimodal-with-STN and unimodal sCT when misalignment exceeds roughly the αa=1 range used here.
  • Further ablation could test whether the STN helps mainly by improving the CT signal fed into the U-Net or by regularizing the network's spatial assumptions; the registration-loss weight makes this separation testable.
  • For clinical adoption, the method could be evaluated against a conventional external rigid registration followed by the same fusion network, to quantify how much end-to-end training contributes beyond pre-alignment alone.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes an end-to-end multimodal synthetic CT (sCT) pipeline that combines a 3D U-Net with a Spatial Transformer Network (STN) to register a preoperative CT to an intraoperative CBCT before early fusion and reconstruction. The STN predicts a 12-parameter affine transformation, and the model is trained with a weighted combination of MAE, 1-SSIM, and VGG-based perceptual losses. The authors evaluate on one synthetic dataset (CBCTLiTS) with controlled CBCT quality and affine misalignment, and on two clinical datasets (Pancreatic-CT-CBCT-SEG and SynthRAD2023). They report that the proposed MM+STN method outperforms the unimodal and multimodal baselines in 79/90 settings, with the largest gains for low CBCT quality and moderate misalignment. The paper also makes code publicly available.

Significance. If the central claim holds, the paper provides a useful incremental extension of prior multimodal sCT work [14] by explicitly modeling affine misalignment with a learnable registration module, supported by a controlled synthetic benchmark and public code with four random splits. The controlled study of CBCT quality and alignment is a strength, and the reproducibility measures (public code, standard deviations over splits) are commendable. However, the significance is limited by the narrow affine-only perturbation protocol used for the clinical datasets and by mixed results on SynthRAD, which are acknowledged but not fully reconciled with the abstract's global claim. The paper would be a reasonable contribution to a specialist venue, provided the generalizability claims are scaled to the evidence.

major comments (4)
  1. [Section 2.1 and Section 4] The claim of 'strong generalizability' to real-world clinical datasets is not supported because every dataset is unaligned with the same TorchIO RandomAffine procedure controlled by αa, and the STN predicts only 12 affine parameters. SynthRAD is explicitly rigidly registered before this synthetic perturbation, so neither clinical dataset provides an out-of-distribution test for non-rigid deformations, respiratory motion, or truncation artifacts, which are the clinically relevant forms of CBCT-CT misalignment. I recommend either adding an experiment with non-rigid perturbations (e.g. B-spline or demon-based deformation) or softening the generalizability conclusion to affine misalignment only.
  2. [Table 1 and Section 2.1] The pancreas dataset is described as including 'naturally occurring misalignment', yet Table 1 reports a CT-only baseline of 0.000±0.000 at αa=0 for all metrics, implying the CT input is exactly equal to the ground-truth CT. This is difficult to reconcile with natural misalignment and suggests that the preprocessing pipeline may have removed or ignored the real misalignment. Please clarify how the pancreatic dataset's native misalignment is handled and whether the reported αa=0 condition corresponds to perfectly aligned pairs.
  3. [Table 2 and Section 3] On the SynthRAD dataset, the proposed MM+STN method worsens MAE in four of the five αa settings (e.g. 0.120→0.127 at αa=1, 0.111→0.114 at αa=0.25) and worsens 1-SSIM in four of five settings (e.g. 0.136→0.142 at αa=1, 0.119→0.146 at αa=0.5). The text concedes the results are 'inconclusive', but these cases are still counted in the 79/90 aggregate improvements. Since sCT quality is primarily quantified by MAE and 1-SSIM, the abstract's claim that integrating registration improves sCT quality is overstated. Please report the 79/90 breakdown by dataset and metric and adjust the wording accordingly.
  4. [Tables 1-3] No significance testing or confidence intervals are reported. With only four random splits, many of the observed differences fall within one standard deviation of the baseline (e.g. pancreas αa=0.25 MAE: 0.105±0.038 vs 0.094±0.028; SynthRad αa=0.125 1-SSIM: 0.128±0.109 vs 0.130±0.114). To support the claim of improvement, the authors should provide paired significance tests (e.g. Wilcoxon signed-rank or paired t-test) or otherwise quantify the fraction of settings where the difference exceeds the noise level.
minor comments (5)
  1. [Section 2] The perceptual loss uses a 2D VGG16 network on 3D volumes, but the text does not specify how the 3D input is converted to 2D slices. Please clarify whether features are computed per axial slice, in all three orientations, or using some other scheme.
  2. [Introduction and Section 4] There are several typos, including 'preoperativce' in the Introduction, 'STM+MM' instead of 'STN+MM' in Section 3, and 'noticable' in Section 4. These should be corrected.
  3. [Figure 2] The figure caption and axis annotations are garbled (e.g. 'perfect alignment44120128'), making the visualization of the unalignment levels difficult to interpret. Please clean up the caption and axis labels.
  4. [Section 2.1] The sentence 'All datasets were unaligned in the same way' conflicts with the statement that the pancreas dataset has naturally occurring misalignment. Please describe the exact preprocessing for each dataset, including whether the natural misalignment is retained, corrected, or augmented.
  5. [Section 3] The sentence 'Overall, CT-only baselines show little difference between MM+STN and MM' is ambiguous. It should specify whether this refers to the CT-only column values or to the comparisons against the CT-only baseline.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the sCT improvement is an empirical, held-out comparison; self-citations are not load-bearing.

full rationale

Walk of the derivation chain: (1) The method trains an STN plus U-Net with a composite loss; the only direct supervision of the STN is the auxiliary registration term 10^-3*MSE(VCT, Y). This term supervises an intermediate alignment, not the reported sCT result. (2) The central claim—that adding the STN improves sCT quality in 79/90 settings—is established by comparing MAE/1-SSIM/perceptual metrics on held-out test splits under identical data splits and synthetic misalignments. The evaluation metric is the generated sCT versus the aligned CT; no reported number is defined as the training loss or as a fitted parameter. (3) The multimodal baseline [14] is the authors' predecessor, but it is reimplemented under the same conditions, so the comparison functions as an empirical ablation, not as an imported conclusion. (4) The CT-only baseline attains zero error at alpha_a=0 because its input equals its target; this is a sanity-check construction, and the paper does not present it as a predicted result. (5) The affine-only unalignment and application of the same synthetic unalignment to clinical datasets are external-validity limitations, not circular reductions. No uniqueness theorem or ansatz is smuggled in via self-citation; the STN is a standard component cited to Jaderberg et al. Overall, no load-bearing step reduces to its own inputs.

Assumptions & free parameters 4 free parameters · 3 assumptions · 0 invented entities

The method introduces no new physical entities. The free parameters are loss weights and the registration loss scale, all chosen by hand. The load-bearing assumptions are the affine-misalignment proxy, the transferability of ImageNet features to 3D medical volumes, and the adequacy of split-averaged metrics without significance testing.

free parameters (4)
  • Loss weight alpha_1 (MAE) = 0.2
    Chosen empirically from validation metrics and qualitative assessment; balances voxel fidelity.
  • Loss weight alpha_2 (1-SSIM) = 0.1
    Chosen empirically; adds local structural coherence and contrast preservation.
  • Loss weight alpha_3 (perceptual) = 0.7
    Chosen empirically; prioritized for sharpness and anatomical detail, but required the MAE term to avoid hallucinated structures.
  • Registration loss weight = 1e-3
    Hand-set small multiplier on MSE between moved CT and aligned CT; no tuning study reported.
assumptions (3)
  • domain assumption Random affine unalignment with parameter alpha_a is a valid proxy for clinical CBCT-CT misalignment.
    Section 2.1; all controlled conclusions about alignment effects and the synthetic dataset are built on this.
  • domain assumption ImageNet-pretrained 2D VGG16 features are meaningful for 3D medical CT volumes.
    Section 2, perceptual loss; no validation or implementation detail for 3D input.
  • domain assumption Average metrics over four random splits without significance testing are sufficient to compare methods.
    Section 2.2 and Section 3; many differences within one standard deviation.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Enhancing Synthetic CT from CBCT via Multimodal Fusion and End-To-End Registration." pith.science (2026). https://pith.science/paper/DMRRRV6S

@misc{pith2026250706067,
  author       = {Pith},
  title        = {Pith review of: Enhancing Synthetic CT from CBCT via Multimodal Fusion and End-To-End Registration},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/DMRRRV6S}},
  note         = {Machine review of arXiv:2507.06067}
}
read the original abstract

Cone-Beam Computed Tomography (CBCT) is widely used for intraoperative imaging due to its rapid acquisition and low radiation dose. However, CBCT images typically suffer from artifacts and lower visual quality compared to conventional Computed Tomography (CT). A promising solution is synthetic CT (sCT) generation, where CBCT volumes are translated into the CT domain. In this work, we enhance sCT generation through multimodal learning by jointly leveraging intraoperative CBCT and preoperative CT data. To overcome the inherent misalignment between modalities, we introduce an end-to-end learnable registration module within the sCT pipeline. This model is evaluated on a controlled synthetic dataset, allowing precise manipulation of data quality and alignment parameters. Further, we validate its robustness and generalizability on two real-world clinical datasets. Experimental results demonstrate that integrating registration in multimodal sCT generation improves sCT quality, outperforming baseline multimodal methods in 79 out of 90 evaluation settings. Notably, the improvement is most significant in cases where CBCT quality is low and the preoperative CT is moderately misaligned.

Figures

Figures reproduced from arXiv: 2507.06067 by the authors.

Figure 1
Figure 1. Proposed 3D, (early fusion) multimodal sCT model, based on U-Net. Specifically, the model begins with an STN component, followed by an en￾coder consisting of three double convolutional blocks, each with 3×3×3 kernels, connected by 3D max pooling layers. The latent space contains one additional double convolutional block, followed by a decoder that mirrors the encoder struc￾ture. Each encoder block is connected to it… view at source ↗
Figure 2
Figure 2. Sample results, showing the original CT as well as synthetically unaligned version of the same CT (CBCTLiTS dataset). Four different volumes are shown with increasing αa (unalignment), displaying rotation, scaling and minor translation. 2.1 Dataset To investigate the effect of the STN, three datasets, one synthetic and two real￾world datasets are used: 1. CBCT Liver Tumor Segmentation Benchmark (CBCTLiTS) dataset [1… view at source ↗
Figure 3
Figure 3. Qualitative comparison of sCT reconstructions on the CBCTLiTS dataset. Rows correspond to different CBCT quality levels (top: low, bottom: high). Columns compare methods: a unimodal CBCT baseline (left) and multimodal reconstructions conditioned on preoperative CTs with increasing alignment (right). Images are nor￾malized for visualization and shown at increased zoom to better highlight anatomical structures. Result… view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

18 extracted references · 17 canonical work pages

  1. [14]

    Enhancing Synthetic CT from CBCT via Multimodal Fusion: A Study on the Impact of CBCT Quality and Alignment

    Tschuchnig, M., Lamminger, L., Steininger, P., Gadermayr, M.: Enhancing syn- thetic ct from cbct via multimodal fusion: A study on the impact of cbct quality and alignment. arXiv preprint arXiv:2506.08716 (2025)

  2. [1]

    IEEE Transactions on Radiation and Plasma Medical Sciences (2025)

    Altalib, A., McGregor, S., Li, C., Perelli, A.: Synthetic ct image generation from cbct: A systematic review. IEEE Transactions on Radiation and Plasma Medical Sciences (2025)

  3. [2]

    Medical Physics47(3), 1115–1125 (2020)

    Chen, L., Liang, X., Shen, C., Jiang, S., Wang, J.: Synthetic ct generation from cbct images via deep learning. Medical Physics47(3), 1115–1125 (2020)

  4. [3]

    Physics in Medicine & Biology 66(11), 115019 (2021)

    Chen, L., Liang, X., Shen, C., Nguyen, D., Jiang, S., Wang, J.: Synthetic ct gen- eration from cbct images via unsupervised deep learning. Physics in Medicine & Biology 66(11), 115019 (2021)

  5. [4]

    In: Medical Image Computing and Computer-Assisted Intervention

    Çiçek, Ö., Abdulkadir, A., Lienkamp, S.S., Brox, T., Ronneberger, O.: 3d u-net: learning dense volumetric segmentation from sparse annotation. In: Medical Image Computing and Computer-Assisted Intervention. pp. 424–432 (2016)

  6. [5]

    In: IEEE Conference on Computer Vision and Pattern Recognition

    Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: Imagenet: A large- scale hierarchical image database. In: IEEE Conference on Computer Vision and Pattern Recognition. pp. 248–255 (2009)

  7. [6]

    Scientific Data 9(1), 637 (2022)

    Hong, J., Reyngold, M., Crane, C., Cuaron, J., Hajj, C., Mann, J., Zinovoy, M., Yorke, E., LoCastro, E., Apte, A.P., et al.: Ct and cone-beam ct of ablative radi- ation therapy for pancreatic cancer with expert organ-at-risk contours. Scientific Data 9(1), 637 (2022)

  8. [7]

    Advances in Neural Information Processing Systems28 (2015)

    Jaderberg, M., Simonyan, K., Zisserman, A., et al.: Spatial transformer networks. Advances in Neural Information Processing Systems28 (2015)

Show all 18 references
  1. [8]

    In: European Conference on Computer Vision–ECCV

    Johnson, J., Alahi, A., Fei-Fei, L.: Perceptual losses for real-time style transfer and super-resolution. In: European Conference on Computer Vision–ECCV. pp. 694–711 (2016)

  2. [9]

    Radiation Oncology 11, 1–10 (2016)

    Oechsner, M., Chizzali, B., Devecka, M., Combs, S.E., Wilkens, J.J., Duma, M.N.: Registration uncertainties between 3d cone beam computed tomography and dif- ferent reference ct datasets in lung stereotactic body radiation therapy. Radiation Oncology 11, 1–10 (2016)

  3. [10]

    In: Medical Image Comput- ing and Computer-Assisted Intervention

    Podobnik, G., Strojan, P., Peterlin, P., Ibragimov, B., Vrtovec, T.: Multimodal ct and mr segmentation of head and neck organs-at-risk. In: Medical Image Comput- ing and Computer-Assisted Intervention. pp. 745–755 (2023)

  4. [11]

    Otolaryngology—Head and Neck Surgery134(5), 801–808 (2006)

    Rafferty,M.A.,Siewerdsen,J.H.,Chan,Y.,Daly,M.J.,Moseley,D.J.,Jaffray,D.A., Irish, J.C.: Intraoperative cone-beam ct for guidance of temporal bone surgery. Otolaryngology—Head and Neck Surgery134(5), 801–808 (2006)

  5. [12]

    In: International Conference on Learning Representations (2015)

    Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale im- age recognition. In: International Conference on Learning Representations (2015)

  6. [13]

    Medical Physics50(7), 4664–4674 (2023)

    Thummerer, A., van der Bijl, E., Galapon Jr, A., Verhoeff, J.J., Langendijk, J.A., Both, S., van den Berg, C.N.A., Maspero, M.: Synthrad2023 grand chal- lenge dataset: Generating synthetic ct for radiotherapy. Medical Physics50(7), 4664–4674 (2023)

  7. [15]

    In: BVM Workshop

    Tschuchnig, M.E., Coste-Marin, J., Steininger, P., Gadermayr, M.: Multi-task learning to improve semantic segmentation of cbct scans using image reconstruc- tion. In: BVM Workshop. pp. 243–248 (2024) Enhancing Synthetic CT via Multimodal Fusion and Registration 11

  8. [16]

    In: Eurographics Work- shop on Visual Computing for Biology and Medicine (2024)

    Tschuchnig, M.E., Steininger, P., Gadermayr, M.: CBCTLiTS: A Synthetic, Paired CBCT/CT Dataset For Segmentation And Style Transfer. In: Eurographics Work- shop on Visual Computing for Biology and Medicine (2024)

  9. [17]

    Medical Physics (2024)

    Wei, C., Albrecht, J., Rit, S., Laurendeau, M., Thummerer, A., Corradini, S., others, Landry, G.: Reduction of cone-beam ct artifacts in a robotic cbct device using saddle trajectories with integrated infrared tracking. Medical Physics (2024)

  10. [18]

    In: IEEE Conference on Computer Vision and Pattern Recognition

    Zhao, Z., Bai, H., Zhang, J., Zhang, Y., Zhang, K., Xu, S., Chen, D., Timofte, R., Van Gool, L.: Equivariant multi-modality image fusion. In: IEEE Conference on Computer Vision and Pattern Recognition. pp. 25912–25921 (2024) 12 M. Tschuchnig et al. T able 3. Evaluation results...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.