Improved MR to CT synthesis for PET/MR attenuation correction using Imitation Learning

Brian Hutton; David Atkinson; Kerstin Kl\"aser; Kris Thielemans; M Jorge Cardoso; Pawel Markiewicz; Sebastien Ourselin; Thomas Varsavsky; Tom Vercauteren

REVIEW 3 major objections 3 minor 12 references

Improved MR to CT synthesis for PET/MR attenuation correction using Imitation Learning

T0 review · 3 major / 3 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Optimizing PET residuals, not CT similarity, improves MR-to-CT synthesis for PET/MR attenuation correction.

desk verdict A genuinely different MR-to-CT synthesis idea for PET/MR attenuation correction, but the slice-level split and unvalidated imitation network mean the headline PET improvement isn't established. read the letter →

arxiv 1908.08431 v2 pith:2TIM4IE6 submitted 2019-08-21 eess.IV cs.CVcs.LGphysics.med-ph

Kerstin Kläser , Thomas Varsavsky , Pawel Markiewicz , Tom Vercauteren , David Atkinson , Kris Thielemans , Brian Hutton , M Jorge Cardoso

show 1 more author

Sebastien Ourselin

This is my paper · ORCID

classification eess.IVcs.CVcs.LGphysics.med-ph

keywords MRtoCTsynthesisPETattenuationcorrectionimitationlearningmulti-hypothesispseudoPET/MRIdeep

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper contends that the usual way of judging MR-to-CT synthesis—how closely the pseudo CT matches the true CT—is the wrong target when the pseudo CT is meant to be used as an attenuation map in PET/MR imaging. The authors build a two-stage framework: a network that predicts several plausible pseudo CTs, and a second 'imitation' network that learns to predict the PET reconstruction error that any given pCT would produce. The first network is then retrained using this learned error metric alongside the standard CT error. On a held-out test set drawn from 400 paired MR/CT/PET slices, the proposed pseudo CTs have a higher CT mean absolute error (69.68 HU) than the baseline (66.25 HU), yet yield a significantly lower PET error (115.41 versus 140.76, paired t-test p<$10^{-4}$). The take-away is that downstream task performance, not intermediate image fidelity, should drive synthesis.

What carries the argument

The central machinery is a three-phase training schedule. In phase one, a convolutional network is trained with a Winner-Takes-All loss to output several plausible pCTs from one MR, so each output head specialises in different anatomical features such as skull thickness or bone density. In phase two, a second network is trained, with the first network frozen, to take a CT and a pCT as input and predict the absolute residual between the true PET and the PET reconstructed using that pCT as the attenuation map; training data comes from a simulated PET acquisition and reconstruction pipeline. In phase three, the first network is retrained with a loss that is an equal combination of the CT L2 error and the learned PET-residual metric, which acts as a differentiable imitation of the reconstruction process. The multi-hypothesis Winner-Takes-All scheme is what allows the generator to maintain several plausible solutions rather than a single risk-minimising average.

What would settle it

Compute the imitation network's predicted PET residual maps on the retrained pCTs and compare them to the actual simulated PET residual maps; if the predictions are substantially inaccurate or biased, the improvement in pPET mean absolute error cannot be attributed to the learned metric.

Watch

Extended reading notes

Core claim

The paper's central discovery is that minimising the intensity-wise error between a synthetic CT and the true CT is not a good surrogate for minimising the error in the PET image that will be reconstructed from that synthetic CT. A small, localised difference in CT attenuation can produce a large error in the reconstructed PET. The authors therefore propose a differentiable proxy for PET reconstruction: a convolutional network trained on pairs of (CT, pCT) to predict the residual between the gold-standard PET and the pseudo-PET reconstructed with each pCT. When this proxy is used as a loss term in retraining the multi-hypothesis pCT generator, the generator learns pCTs that score worse on CT mean absolute error but better on PET mean absolute error by a statistically significant margin. The paper also reports that sampling multiple pCT hypotheses gives a more reliable estimate of PET uncertainty than Monte-Carlo dropout.

Load-bearing premise

The imitation network, trained on only 60 pCT/pPET pairs from 20 slices, remains accurate on the pCTs produced after the first network is retrained, even though those retrained pCTs were not part of its training distribution.

Editorial extensions

If this is right

Evaluation of pCT synthesis should include downstream PET error, not just CT intensity metrics, because the two can disagree.
A learned differentiable proxy for PET reconstruction can be used to optimise the pCT generator without needing to reconstruct every training sample.
Multi-hypothesis sampling gives a wider and more reliable distribution of pPET intensities than Monte-Carlo dropout when estimating PET uncertainty.
The framework produces an average reduction in pPET mean absolute error from about 141 to about 115 arbitrary units on the held-out test data.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The same imitation-loss idea could transfer to radiotherapy planning, where synthetic CTs are used for dose calculation and the real metric is dose error, not CT intensity error.
Because the imitation network is trained on a specific PET reconstruction simulation, its benefit may shrink if the clinical reconstruction pipeline differs from the simulation; testing across reconstruction parameters would clarify this.
The multi-hypothesis outputs could serve as a built-in uncertainty estimate for the pCT itself, potentially flagging slices where the downstream PET is unreliable.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

First thing you should know: this paper proposes something genuinely different for MR-to-CT synthesis in PET/MR. Instead of minimizing CT error directly, the authors generate multiple pCT hypotheses with a winner-takes-all loss, train a second CNN to imitate the PET reconstruction residual from a CT/pCT pair, then retrain the first network with that learned metric as part of its loss. The result they report—lower pPET MAE (115 vs 141) despite slightly higher CT MAE (70 vs 66)—is exactly what you'd want to see if the idea works. The authors earn credit for evaluating on the downstream endpoint and for illustrating that localized CT differences produce disproportionately large PET residuals.

The soft spots are not minor. Section 2.4 says the 70/10/20 split was applied to whole images—meaning slices, not patients. With 20 patients and 20 central slices each, the test set almost certainly contains slices from every patient, and adjacent central slices from the same patient are highly correlated. The paired t-test across slices is pseudoreplication; a patient-level split could easily erase the p<1e-4. That is load-bearing because the entire claim is cross-patient generalization.

The second issue is the imitation network. It was trained on 60 pairs from 20 slices and never validated. Once the first network is retrained using this network's outputs as a loss, the pCTs can move outside the distribution g_psi was trained on, and nothing in the paper checks whether g_psi is still accurate there. This matters because g_psi is the mechanism, not a detail.

A smaller but real caveat: the PET data were simulated because raw PET data were unavailable. The authors are transparent about this, but the reported 18% improvement is in a simulation, not in real reconstructed PET.

Who is this for: people working on PET/MR attenuation correction, and more broadly anyone designing task-specific losses for image translation. The idea is worth testing properly; the reported evidence is not. If I were editor, I would send it to review, but with the clear expectation that the authors redo the split at patient level, validate the surrogate metric, and temper the claims until those analyses exist. Desk rejection would be too harsh; acceptance without major revision would be unjustified.

Referee Report

3 major / 3 minor

Summary. The manuscript proposes a two-stage deep learning method for MR-to-CT synthesis in PET/MR attenuation correction. A multi-hypothesis HighResNet first generates M candidate pCTs; a second 'imitation' network learns to predict PET reconstruction residual maps from CT/pCT pairs; the first network is then retrained with a loss combining the CT L2 error and the imitation metric. The authors report on 400 slices from 20 patients that the proposed imitation learning yields a higher CT MAE (69.68 HU) than a baseline HighResNet (66.25 HU) but a significantly lower PET MAE (115.41 vs 140.76 arbitrary units; paired t-test p<10^-4). They also compare multi-hypothesis sampling with MC dropout for uncertainty estimation.

Significance. If the claimed downstream PET improvement were robustly demonstrated, the paper would make a useful contribution to MR-based attenuation correction by explicitly optimizing for PET-relevant error rather than CT fidelity, and by introducing a learned differentiable surrogate for the PET reconstruction process. The multi-hypothesis formulation and the comparison with MC dropout are also interesting. However, the current evidence is undermined by methodological concerns in the evaluation protocol and by the lack of validation of the surrogate metric network; the central claim that minimizing the learned metric improves final PET reconstruction remains to be established.

major comments (3)

[Section 2.3 and Section 3] The 70/10/20 split is described as being performed on 'whole images', which in this paper are the 400 extracted 2D slices from 20 patients. The manuscript does not state that the split is patient-wise. Since the 20 central slices from a single patient are highly correlated in anatomy and intensity, a random slice-level split places slices from the same patient in both training and test sets. The reported paired t-test on the test slices then treats within-patient correlated observations as independent, which is pseudoreplication. Consequently, the central quantitative claim—that imitation learning significantly reduces pPET MAE (p<10^-4)—does not demonstrate cross-patient generalization. The authors should repeat the evaluation with a patient-level split or use a cluster-adjusted statistical test.
[Section 2.2] The imitation network g_psi is trained on only 60 pCT/pPET pairs derived from 20 slices, and no validation of g_psi is reported (e.g., accuracy of its predicted PET residual maps on held-out patients). After the first network is retrained using g_psi as a loss term, the resulting pCTs are likely outside the distribution of the 60 training pairs. Without evidence that g_psi remains accurate on pCTs produced by the retrained network, the proposed mechanism for the PET improvement is unsupported. The authors should provide a quantitative evaluation of g_psi on an independent test set and demonstrate that minimizing the learned metric correlates with true PET residuals.
[Section 2.2] The combined loss for the first network is written as L(x_i, y_i, z_i) = min_m [g_psi(f_phi(x_i), y_i), z_i]. This expression is not a well-defined scalar objective: g_psi outputs an error map, z_i is the ground-truth residual map, and the bracketed notation is ambiguous. The text later states that the first network is retrained using 'both the CT L2-loss and the metric loss in equal proportions', but the exact functional form and weighting are not specified. Since this loss is the core of the proposed method, the authors must define it precisely to make the method reproducible and the optimization objective unambiguous.

minor comments (3)

[Section 3] The description of data extraction states that '20 central slices per volume' were used, resulting in 400 images, but it is unclear how these slices were selected across patients (e.g., whether the same anatomical locations were used). This matters for interpreting the slice-level split and the extent of correlation in the test set.
[Section 4] The PET MAE values are reported in arbitrary units ('a.u.'), but the manuscript does not explain how these units are defined or normalized. Please clarify the scale of the PET intensities and whether the MAE is computed on absolute activity values or on some standardized scale.
[Section 2.4] Several implementation details are missing, including the number of hypotheses M, the exact weighting between the CT L2 loss and the metric loss during retraining, and the convergence criteria. Reporting these would improve reproducibility.

Circularity Check

0 steps flagged · score 0.0 of 10

No constructional circularity: the claimed pPET improvement is assessed with actual PET reconstruction, not with the fitted surrogate loss.

full rationale

The paper's derivation chain is a three-stage training loop: (1) a multi-hypothesis HighResNet f is trained with an L2 winner-takes-all loss; (2) a second network g_psi is trained on pCTs produced by the frozen f to predict the PET residual between the true PET and the pPET reconstructed from each pCT; (3) f is retrained using a combination of the CT L2-loss and g_psi as a learned metric. The final quantitative claim is not obtained by evaluating g_psi on the retrained pCTs. Instead, the pCTs from the retrained network are used as mu-maps in a real PET forward-projection and reconstruction simulation, and the reported pPET MAE is computed against the true PET reconstructed from the CT. The predicted quantity is therefore not defined as the fitted metric output, and no equation in the paper reduces the final result to the training input by construction. The main weaknesses are correctness risks rather than circularity: the 70/10/20 split is performed on slices rather than patients, so the paired t-test may pseudoreplicate correlated slices from the same subject; and g_psi is trained on only 60 pCT/pPET pairs from 20 slices and is never validated, so the retrained f could shift outside the distribution where g_psi is accurate. These are empirical and statistical concerns, not definitional circularity. The self-citations (HighResNet [9], NiftyNet [10], NiftyPET [12]) are implementation tools or the baseline architecture and are not load-bearing to the central PET-improvement claim. No circular step can be exhibited with a specific reduction.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central claim rests on several domain assumptions about CT-to-mu conversion and PET simulation, plus an ad hoc choice of 20 slices for the imitation network. The only hand-set free parameters are the loss weighting, the number of hypotheses, and the data split ratio. No new physical entities are introduced.

free parameters (3)

Loss weighting between CT L2 and metric loss = 0.5/0.5 (equal proportions)
Chosen by hand, not optimized; the relative weighting determines the trade-off between CT fidelity and PET residual minimization.
Number of hypotheses M = not specified
Number of output heads in the multi-hypothesis network; chosen by hand, not swept or reported.
Train/validation/test split ratio = 70/10/20
Choosing a slice-level split rather than patient-level split directly affects the measured test performance and the significance test.

assumptions (4)

domain assumption CT Hounsfield units bi-linearly approximate linear attenuation coefficients (mu)
Standard assumption in PET/MR attenuation correction, stated in Section 1.
domain assumption Simulated PET forward projection and reconstruction via NiftyPET faithfully models true PET acquisition
The imitation network's training targets and the evaluation metrics both rely on this simulation; if the simulation is inaccurate, the claimed benefit is an artifact.
domain assumption MR and CT images are correctly registered after rigid and low-DOF non-rigid alignment
Misalignment between MR and CT would corrupt the paired training data; stated in Section 3.
ad hoc to paper The 20 slices used to train the imitation network are representative of the full dataset
The paper selects 20 slices to build the metric network without justification of representativeness; Section 3.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Improved MR to CT synthesis for PET/MR attenuation correction using Imitation Learning." pith.science (2026). https://pith.science/paper/2TIM4IE6

@misc{pith2026190808431,
  author       = {Pith},
  title        = {Pith review of: Improved MR to CT synthesis for PET/MR attenuation correction using Imitation Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2TIM4IE6}},
  note         = {Machine review of arXiv:1908.08431}
}

abstract

The ability to synthesise Computed Tomography images - commonly known as pseudo CT, or pCT - from MRI input data is commonly assessed using an intensity-wise similarity, such as an L2-norm between the ground truth CT and the pCT. However, given that the ultimate purpose is often to use the pCT as an attenuation map ($\mu$-map) in Positron Emission Tomography Magnetic Resonance Imaging (PET/MRI), minimising the error between pCT and CT is not necessarily optimal. The main objective should be to predict a pCT that, when used as $\mu$-map, reconstructs a pseudo PET (pPET) which is as close as possible to the gold standard PET. To this end, we propose a novel multi-hypothesis deep learning framework that generates pCTs by minimising a combination of the pixel-wise error between pCT and CT and a proposed metric-loss that itself is represented by a convolutional neural network (CNN) and aims to minimise subsequent PET residuals. The model is trained on a database of 400 paired MR/CT/PET image slices. Quantitative results show that the network generates pCTs that seem less accurate when evaluating the Mean Absolute Error on the pCT (69.68HU) compared to a baseline CNN (66.25HU), but lead to significant improvement in the PET reconstruction - 115a.u. compared to baseline 140a.u.

Figures

Figures reproduced from arXiv: 1908.08431 by the authors.

**Figure 1.** a) The ground truth CT, b) the predicted pseudo CT, c) the absolute residual between true and pseudo CT, and d) the absolute residual between PETs reconstructed using the CT and synthetic CT as attenuation maps. Note that small and very localised difference in the CT (c) result in large PET residuals (d). We argue that algorithms should be optimising for PET residuals (d) and not for CT residuals (c). cycleGAN and t… view at source ↗

**Figure 2.** Yellow box: semantic regression. Net1 takes MR images as inputs and predicts multiple pCT realisations by minimising a combination of the L2-loss between ground truth CT and pCT (L2-loss CT) and a learned metric loss (L2-loss ML). In the first stage only L2-loss CT is considered and L2-loss ML is weighted to zero. Purple box: imitation network. Net2 takes pCTs and corresponding CTs as input and predicts the error be… view at source ↗

**Figure 3.** Qualitative results. From top to bottom: Ground-truth, baseline (HighResNet), and imitation learning. From left to right: CT, pCT-CT residuals, PET, pPET-PET residuals. As expected, we note that MAE in the pCT generated with the proposed imitation learning is higher than the baseline, but the resulting pPET error is significantly lower for the proposed method. the pseudo PET only in the brain region by masking out … view at source ↗

Figures from the paper (1 more)

**Figure 4.** Figure 4: PET intensities (first column), variance (middle column) and Z-score (right column) of ground truth PET (top row) compared to pPET intensities reconstructed with pCTs from Monte Carlo (MC) dropout sampling (middle row) and pCTs from multi-hypothesis sampling (bottom ro…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

12 extracted references · 11 canonical work pages

[1]

Volume 38., Elsevier (2008) 199–208

Pichler, B.J., et al.: Positron emission tomography/magnetic resonance imaging: the next generation of multimodality imaging? In: Seminars in nuclear medicine. Volume 38., Elsevier (2008) 199–208

work page 2008
[2]

Neu- roimage 147 (2017) 346–359

Ladefoged, C.N., et al.: A multi-centre evaluation of eleven clinically feasible brain pet/mri attenuation correction techniques using a large cohort of patients. Neu- roimage 147 (2017) 346–359

work page 2017
[3]

IEEE TMI 33(12) (2014) 2332–2341

Burgos, N., et al.: Attenuation correction synthesis for hybrid pet-mr scanners: application to brain studies. IEEE TMI 33(12) (2014) 2332–2341

work page 2014
[4]

arXiv preprint arXiv:1703.10593 (2017)

Zhu, J.Y., et al.: Unpaired image-to-image translation using cycle-consistent ad- versarial networks. arXiv preprint arXiv:1703.10593 (2017)

arXiv 2017
[5]

In: SASHIMI, Springer (2017) 14–23

Wolterink, J.M., et al.: Deep mr to ct synthesis using unpaired data. In: SASHIMI, Springer (2017) 14–23

work page 2017
[6]

In: Deep Learning in Medical Image Analysis and Multimodal Learning for Clinical Decision Support, Springer International Publishing (2018) 174–182

Yang, H., et al.: Unpaired brain mr-to-ct synthesis using a structure-constrained cyclegan. In: Deep Learning in Medical Image Analysis and Multimodal Learning for Clinical Decision Support, Springer International Publishing (2018) 174–182

work page 2018
[7]

Sensors 19(10) (2019) 2361

Jin, C.B., et al.: Deep ct to mr synthesis using paired and unpaired data. Sensors 19(10) (2019) 2361

work page 2019
[8]

In: Proceedings of the IEEE ICCV

Rupprecht, C., et al.: Learning in an uncertain world: Representing ambiguity through multiple hypotheses. In: Proceedings of the IEEE ICCV. (2017) 3591– 3600

work page 2017

Show all 12 references

[9]

In: IPMI, Springer (2017) 348–360

Li, W., et al.: On the compactness, eﬃciency, and representation of 3d convolu- tional networks: brain parcellation as a pretext task. In: IPMI, Springer (2017) 348–360

2017
[10]

CoRR abs/1709.03485 (2017)

Gibson, E., et al.: Niftynet: a deep-learning platform for medical imaging. CoRR abs/1709.03485 (2017)

2017 arXiv
[11]

Computer methods and programs in biomedicine 98(3) (2010) 278–284

Modat, M., et al.: Fast free-form deformation using graphics processing units. Computer methods and programs in biomedicine 98(3) (2010) 278–284

2010
[12]

Neuroinformatics 16(1) (2018) 95–115

Markiewicz, P.J., et al.: Niftypet: a high-throughput software platform for high quantitative accuracy and precision pet imaging and analysis. Neuroinformatics 16(1) (2018) 95–115

2018

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.

[1] [1]

Volume 38., Elsevier (2008) 199–208

Pichler, B.J., et al.: Positron emission tomography/magnetic resonance imaging: the next generation of multimodality imaging? In: Seminars in nuclear medicine. Volume 38., Elsevier (2008) 199–208

work page 2008

[2] [2]

Neu- roimage 147 (2017) 346–359

Ladefoged, C.N., et al.: A multi-centre evaluation of eleven clinically feasible brain pet/mri attenuation correction techniques using a large cohort of patients. Neu- roimage 147 (2017) 346–359

work page 2017

[3] [3]

IEEE TMI 33(12) (2014) 2332–2341

Burgos, N., et al.: Attenuation correction synthesis for hybrid pet-mr scanners: application to brain studies. IEEE TMI 33(12) (2014) 2332–2341

work page 2014

[4] [4]

arXiv preprint arXiv:1703.10593 (2017)

Zhu, J.Y., et al.: Unpaired image-to-image translation using cycle-consistent ad- versarial networks. arXiv preprint arXiv:1703.10593 (2017)

arXiv 2017

[5] [5]

In: SASHIMI, Springer (2017) 14–23

Wolterink, J.M., et al.: Deep mr to ct synthesis using unpaired data. In: SASHIMI, Springer (2017) 14–23

work page 2017

[6] [6]

In: Deep Learning in Medical Image Analysis and Multimodal Learning for Clinical Decision Support, Springer International Publishing (2018) 174–182

Yang, H., et al.: Unpaired brain mr-to-ct synthesis using a structure-constrained cyclegan. In: Deep Learning in Medical Image Analysis and Multimodal Learning for Clinical Decision Support, Springer International Publishing (2018) 174–182

work page 2018

[7] [7]

Sensors 19(10) (2019) 2361

Jin, C.B., et al.: Deep ct to mr synthesis using paired and unpaired data. Sensors 19(10) (2019) 2361

work page 2019

[8] [8]

In: Proceedings of the IEEE ICCV

Rupprecht, C., et al.: Learning in an uncertain world: Representing ambiguity through multiple hypotheses. In: Proceedings of the IEEE ICCV. (2017) 3591– 3600

work page 2017