REVIEW 3 major objections 6 minor 1 cited by
Empirical Bayesian Mixture Models for Medical Image Translation
T0 review · 3 major / 6 minor · reviewed 2026-08-14 · deepseek-v4-flash
Pith's one-line read A probabilistic model of joint brain-scan intensities can predict entirely missing MR contrasts and CT images from one or a few observed contrasts.
desk verdict A solid, interpretable generative alternative to deep learning for medical image translation, but the quantitative evidence is inflated by whole-volume PSNR and the missing-at-random assumption goes untested. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The workhorse is the unified segmentation model recast as a variational Gaussian mixture with Gauss-Wishart priors on each class's mean and precision, and a deformable template defining voxel-wise class probabilities. The key move is treating unobserved channels as missing data: the observed-channel likelihood marginalizes the Gaussian over missing dimensions (Eq. 6), and the expected marginal posterior over missing values, $E[h_d] = \sum_k \tilde{z}_{dk} \tilde{h}_{dk}$ from Eq. (9), is the synthetic image. Population parameters (template and prior hyperparameters) are learned by variational expectation-maximisation across training subjects, which is what makes the model empirical Bayesian.
What would settle it
Take a cohort with naturally truncated fields of view, infer the missing voxels with the trained model, and compare them against the same anatomy imaged in a separate full-coverage scan; if inferred intensities diverge systematically in exactly the regions where coverage is most often cut off, such as the posterior fossa, the missing-at-random assumption fails.
Extended reading notes
Core claim
The central claim is that if a group-wise generative model of brain scans — a variational Gaussian mixture with class labels driven by a deformable template — is allowed to treat unobserved channels as missing data, then the same fitted model that normalizes and segments a subject also outputs the missing images. For each voxel, the posterior over the missing contrasts is a mixture of Gaussians, and its expected value $E[h_d] = \sum_k \tilde{z}_{dk} \tilde{h}_{dk}$ gives the predicted intensities. The authors demonstrate this on three tasks: filling voxels absent because of differing fields of view, predicting entirely missing MR contrasts in all permutations of observed and missing channels, and synthesizing CT from T1w and T2w MR images. They report that the model produces realistic images, that T1w is the most predictive contrast, and that MR-to-CT translation from only eight training subjects reaches a mean PSNR of 25.5 dB.
Load-bearing premise
The whole prediction rests on the assumption that missingness is ignorable: whether a voxel's channel is unobserved must be unrelated to what its value would have been, which the paper never states or tests.
Editorial extensions
If this is right
- Missing fields of view in routine clinical MR scans can be filled in rather than discarded, with mean PSNR above 36 dB when half the field of view is missing.
- An entirely missing MR contrast can be predicted from any combination of remaining contrasts, with T1w the most informative; predicted contrasts are accurate enough to serve as inputs to deep-learning pipelines trained on a fixed contrast.
- CT images can be synthesized from T1w and T2w MR scans after training on as few as eight subjects, offering a radiation-free route to attenuation correction in PET-MR.
- The same fitted model yields segmentation, bias correction, and spatial normalization alongside translation, so downstream analysis needs no separate preprocessing.
Reading between the lines
- The same machinery could be pushed to scanner harmonization: fitting the joint model on multi-site data and generating a template-contrast image for each subject would isolate site-specific intensity shifts from biological variability.
- The missing-at-random assumption, which the paper leaves implicit, can be tested directly by comparing synthetically masked voxels (where the assumption holds by construction) with naturally missing voxels from truncated acquisitions; systematic divergence would indicate bias in real-world applications.
- The posterior variance of the expected marginal could serve as a per-voxel confidence map, flagging regions where the synthesized modality is unreliable — a guardrail that adversarial image translation methods lack.
- A natural extension is to optimize the template on the target modality as well as the intensity hyperparameters; the paper notes that the meninges and bone detail are missed when only intensities are retrained, so joint template-intensity fitting could close that gap.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper presents a generative model for medical image translation based on an empirical Bayesian Gaussian mixture model. The unified segmentation model of Ashburner and Friston is extended to handle missing data, so that a multivariate Gaussian mixture with a deformable tissue template can be trained on partially observed multi-channel images and then used to predict missing modalities (MR contrasts or CT) for unseen subjects. The training procedure is a variational EM algorithm in which subject-specific posteriors are computed with a mean-field approximation and population prior hyper-parameters are estimated from the data. Experiments are reported on three tasks: inferring missing spatial voxels in MR volumes with different fields of view (IXI dataset), inferring entirely missing MR contrasts (IXI), and translating T1w/T2w MRI into CT (RIRE). All quantitative results are given as PSNR computed on held-out subjects. The paper's central claim is that image translation is a by-product of learning a joint intensity distribution, and that the model works with a small number of training subjects.
Significance. If the findings are supported, the proposed model would be a valuable interpretable alternative to deep-learning-based image translation, with the same model providing segmentation, spatial normalization, and missing-data synthesis in a single probabilistic framework. The derivation of the variational updates (Eqs. 12–23) is standard and internally consistent, and the use of held-out subjects for all reported PSNR values is a clear strength. The paper also honestly acknowledges several limitations, including the qualitative comparison with deep learning and the failure to capture detailed bone structure in CT synthesis. However, the quantitative evidence for the headline claim is weakened by the choice of evaluation metric (whole-volume PSNR without excluding background), the absence of baseline comparisons on the same data, and an unstated missingness assumption. These issues are addressable, but they need to be resolved before the central claim is fully convincing.
major comments (3)
- [Section 3, Eq. (24)] The paper explicitly states that "no voxels are excluded when computing the PSNR," and this is the only quantitative metric used. Head MR and CT volumes contain a large fraction of air/background voxels with near-constant intensities, so a Gaussian mixture component with a small-variance background class can score those voxels almost perfectly. This can substantially inflate the mean PSNR values reported in Tables 1 and 2 and the MR-to-CT result in Section 3.2. Please recompute PSNR within a brain mask (or a tissue mask) and report both whole-volume and masked values, so that the claim that the model "does a good job at filling in missing fields of view" is supported by tissue-only accuracy.
- [Section 3, Tables 1 and 2] No baseline comparison is provided on the same held-out data. The PSNR values are reported only for the proposed model, and the reference to prior work in Section 3.1 is a vague statement that results are "close to those previously reported" without citing specific numbers. Similarly, the CT experiment in Section 3.2 states the result is "not on pair with deep learning based techniques" but does not give any quantitative comparison. Please add at least one baseline on the same data splits (for example, a conditional-mean imputation from a complete-case Gaussian mixture, or a published synthesis method) so that the performance gain of the proposed model is demonstrated rather than assumed.
- [Section 2, Eqs. (6)–(7)] The derivation of the observed-channel likelihood and the conditional prediction of missing channels assumes that missingness is ignorable, i.e., the probability that a channel is missing does not depend on the missing intensity value. This assumption is never stated or tested in the manuscript. In clinical data, missingness may be associated with anatomy, pathology, or field-of-view choices, in which case the predicted values from Eq. (9) would be biased. Please state the missingness assumption explicitly and discuss its validity; a sensitivity analysis with simulated non-ignorable missingness would strengthen the claim that the model has clinical utility.
minor comments (6)
- [Eq. (2)] The equation for the categorical prior is not typeset clearly: the denominator is missing a separating space and the definition of a_dk(φ) is deferred to later text. Please revise for readability.
- [Eq. (17)] The subscripts o and m in the expectations E[z_dk x_d]_o, E[z_dk x_d]_m, etc., are not self-explanatory. Define these subscripts (observed and missing parts) before presenting the update equations.
- [Section 3.1] The text says the model is trained on "50 subjects from the IXI dataset" and then specifies "IXI[064-118]" and "IXI[002-063]"; please verify that these indices each correspond to 50 subjects and state the counts explicitly.
- [Section 3.1] The comparison with Chartsias et al. is only qualitative; please include the exact PSNR value from that reference when you say the results are "close", or remove the comparison.
- [Conclusions] There is a typo in "can then be drived to zero" (should be "driven"). Please correct this and any other grammatical errors.
- [Reproducibility] The paper does not mention code or data availability. For a methodological paper, please include a statement about whether the implementation will be released.
Circularity Check
No significant circularity: held-out predictions are genuine forecasts of unseen modalities under a generative model.
full rationale
No circularity found. The derivation chain from Eqs. (6)-(9) is a standard conditional-Gaussian computation under a variational Gaussian mixture: Eq. (6) marginalizes observed channels, Eq. (7) gives the conditional distribution of missing channels, and Eq. (9) forms the posterior predictive mean. The estimator E[hd] = sum_k z~dk h~dk is evaluated on held-out subjects whose missing channels are not used to fit the population hyper-parameters or the subject-level posterior, so the reported PSNR values in Tables 1-2 and the RIRE experiment are genuine predictions rather than fitted inputs renamed as predictions. The paper's reliance on prior work for template update rules, e.g., 'We do not provide update rules for the template (a), as they can be found in [17]', is a normal citation of an established method and is not load-bearing for the central claim, which is demonstrated empirically on unseen subjects. Concerns about whole-volume PSNR including background voxels are a metric-strength issue, not evidence of circularity, and the missing-at-random assumption is an unstated correctness risk, not a circular dependency.
Assumptions & free parameters
free parameters (2)
- number of Gaussian mixture components K =
12
- global class proportions omega =
estimated, not reported
assumptions (6)
- domain assumption Missing channels are missing at random, so the observed likelihood is obtained by marginalizing the Gaussian over missing channels.
- domain assumption Class-conditional intensities are multivariate Gaussian within each of K tissue classes.
- domain assumption The mean-field factorization of the posterior is a good approximation.
- domain assumption A deformable softmax template with global class proportions describes the spatial distribution of classes.
- domain assumption Conjugate Gauss-Wishart priors adequately capture population variability of intensity parameters.
- domain assumption The template learned from healthy IXI subjects transfers to RIRE patients with pathology, thick slices, and different scanners.
Cite this review
Pith. "Pith review of Empirical Bayesian Mixture Models for Medical Image Translation." pith.science (2026). https://pith.science/paper/LE4LGHDE
@misc{pith2026190805926,
author = {Pith},
title = {Pith review of: Empirical Bayesian Mixture Models for Medical Image Translation},
year = {2026},
howpublished = {\url{https://pith.science/paper/LE4LGHDE}},
note = {Machine review of arXiv:1908.05926}
}
read the original abstract
Automatically generating one medical imaging modality from another is known as medical image translation, and has numerous interesting applications. This paper presents an interpretable generative modelling approach to medical image translation. By allowing a common model for group-wise normalisation and segmentation of brain scans to handle missing data, the model allows for predicting entirely missing modalities from one, or a few, MR contrasts. Furthermore, the model can be trained on a fairly small number of subjects. The proposed model is validated on three clinically relevant scenarios. Results appear promising and show that a principled, probabilistic model of the relationship between multi-channel signal intensities can be used to infer missing modalities -- both MR contrasts and CT images.
Figures
Forward citations
Cited by 1 Pith paper
-
A Tool for Super-Resolving Multimodal Clinical MRI
A generative super-resolution model using multi-channel total variation reconstructs 1 mm isotropic images from thick-sliced multimodal clinical MRI, improving downstream age and sex prediction versus interpolation.
Reference graph
Works this paper leans on
-
[1]
Attenuation correction synthesis for hybrid PET-MR scanners,
N. Burgos, M. J. Cardoso, M. Modat, S. Pedemonte, J. Dickson, A. Barnes, J. S. Duncan, D. Atkinson, S. R. Arridge, B. F. Hutton, et al. , “Attenuation correction synthesis for hybrid PET-MR scanners,” in MICCAI, pp. 147–154, Springer, 2013
work page 2013
-
[2]
Regis- tration for correlative microscopy using image analogies,
T. Cao, C. Zach, S. Modla, D. Powell, K. Czymmek, and M. Niethammer, “Regis- tration for correlative microscopy using image analogies,” in WBIR, pp. 296–306, Springer, 2012
work page 2012
-
[3]
MR contrast synthesis for lesion segmentation,
S. Roy, A. Carass, N. Shiee, D. L. Pham, and J. L. Prince, “MR contrast synthesis for lesion segmentation,” in ISBI, pp. 932–935, IEEE, 2010
work page 2010
-
[4]
MRI modalitiy transformation in demon registra- tion,
D.-J. Kroon and C. H. Slump, “MRI modalitiy transformation in demon registra- tion,” in ISBI, pp. 963–966, IEEE, 2009
work page 2009
-
[5]
A. Guimond, A. Roche, N. Ayache, and J. Meunier, “Three-dimensional multi- modal brain warping using the demons algorithm and adaptive intensity correc- tions,” IEEE T. Med. Imaging , vol. 20, no. 1, pp. 58–69, 2001
work page 2001
-
[6]
Automatic CT-ultrasound registration for diagnostic imaging and image-guided intervention,
W. Wein, S. Brunke, A. Khamene, M. R. Callstrom, and N. Navab, “Automatic CT-ultrasound registration for diagnostic imaging and image-guided intervention,” Med. Image Anal. , vol. 12, no. 5, pp. 577–585, 2008
work page 2008
-
[7]
S.-H. Hsu, Y. Cao, K. Huang, M. Feng, and J. M. Balter, “Investigation of a method for generating synthetic CT models from MRI scans of the head and neck for radiation therapy,” Phys. Med. Biol. , vol. 58, no. 23, p. 8419, 2013
work page 2013
-
[8]
Estimating CT image from MRI data using structured random forest and auto-context model,
T. Huynh, Y. Gao, J. Kang, L. Wang, P. Zhang, J. Lian, and D. Shen, “Estimating CT image from MRI data using structured random forest and auto-context model,” IEEE T. Med. Imaging , vol. 35, no. 1, pp. 174–183, 2015
work page 2015
Show all 23 references
-
[9]
Is synthesizing MRI contrast useful for inter-modality analysis?,
J. E. Iglesias, E. Konukoglu, D. Zikic, B. Glocker, K. Van Leemput, and B. Fischl, “Is synthesizing MRI contrast useful for inter-modality analysis?,” in MICCAI, pp. 631–638, Springer, 2013
2013
-
[10]
A compressed sensing approach for MR tissue contrast synthesis,
S. Roy, A. Carass, and J. Prince, “A compressed sensing approach for MR tissue contrast synthesis,” in IPMI, pp. 371–383, Springer, 2011
2011
-
[11]
Multimodal mr syn- thesis via modality-invariant latent representation,
A. Chartsias, T. Joyce, M. V. Giuffrida, and S. A. Tsaftaris, “Multimodal mr syn- thesis via modality-invariant latent representation,” IEEE transactions on medical imaging, vol. 37, no. 3, pp. 803–814, 2017
2017
-
[12]
Medical image synthesis with context-aware generative adversarial networks,
D. Nie, R. Trullo, J. Lian, C. Petitjean, and M. Ruan, “Medical image synthesis with context-aware generative adversarial networks,” pp. 417–425, Springer, 2017
2017
-
[13]
Deep MR to CT synthesis using unpaired data,
J. M. Wolterink, A. M. Dinkla, M. H. Savenije, P. R. Seevinck, C. A. van den Berg, and I. Iˇ sgum, “Deep MR to CT synthesis using unpaired data,” inSASHIMI, pp. 14–23, Springer, 2017
2017
-
[14]
Distribution matching losses can hallucinate features in medical image translation,
J. P. Cohen, M. Luck, and S. Honari, “Distribution matching losses can hallucinate features in medical image translation,” in MICCAI, pp. 529–536, Springer, 2018
2018
-
[15]
Supervised learning from incomplete data via an EM approach,
Z. Ghahramani and M. I. Jordan, “Supervised learning from incomplete data via an EM approach,” in NeurIPS, pp. 120–127, 1994
1994
-
[16]
Unified segmentation,
J. Ashburner and K. J. Friston, “Unified segmentation,” NeuroImage, vol. 26, no. 3, pp. 839–851, 2005
2005
-
[17]
Generative diffeomor- phic modelling of large MRI data sets for probabilistic template construction,
C. Blaiotta, P. Freund, M. J. Cardoso, and J. Ashburner, “Generative diffeomor- phic modelling of large MRI data sets for probabilistic template construction,” NeuroImage, vol. 166, pp. 117–134, 2018
2018
-
[18]
Empirical Bayes: Past, present and future,
B. P. Carlin and T. A. Louis, “Empirical Bayes: Past, present and future,” J. Am. Stat. Assoc., vol. 95, no. 452, pp. 1286–1289, 2000
2000
-
[19]
Variational inference for medical image segmentation,
C. Blaiotta, M. J. Cardoso, and J. Ashburner, “Variational inference for medical image segmentation,” Comput. Vis. Image Und. , vol. 151, pp. 14–28, 2016
2016
-
[20]
C. M. Bishop, Pattern recognition and machine learning . Springer, 2006
2006
-
[21]
Nonlinear markov random fields learned via backpropagation,
M. Brudfors, Y. Balbastre, and J. Ashburner, “Nonlinear markov random fields learned via backpropagation,” in IPMI, pp. 805–817, Springer, 2019
2019
-
[22]
Comparison and evaluation of retrospective intermodality image registration techniques,
J. B. West, . . ., and R. P. Woods, “Comparison and evaluation of retrospective intermodality image registration techniques,” in Medical Imaging 1996: Image Pro- cessing, vol. 2710, pp. 332–348, SPIE, 1996
1996
-
[23]
MRI super-resolution using multi-channel total variation,
M. Brudfors, Y. Balbastre, P. Nachev, and J. Ashburner, “MRI super-resolution using multi-channel total variation,” in MIUA, pp. 217–228, Springer, 2018
2018
Reviewed August 14, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.