Pith. sign in

REVIEW 3 major objections 6 minor 1 cited by

Empirical Bayesian Mixture Models for Medical Image Translation

T0 review · 3 major / 6 minor · reviewed 2026-08-14 · deepseek-v4-flash

Pith's one-line read A probabilistic model of joint brain-scan intensities can predict entirely missing MR contrasts and CT images from one or a few observed contrasts.

desk verdict A solid, interpretable generative alternative to deep learning for medical image translation, but the quantitative evidence is inflated by whole-volume PSNR and the missing-at-random assumption goes untested. read the letter →

arxiv 1908.05926 v1 pith:LE4LGHDE submitted 2019-08-16 eess.IV cs.CV

classification eess.IVcs.CV
keywords medicalimagetranslationGaussianmixturemodelmissingdataempiricalBayesvariationalinferenceMRIsynthesisCTunifiedsegmentation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that medical image translation — synthesizing one imaging modality from another — can be achieved as a by-product of learning a joint probability distribution over multi-channel intensities, rather than by training a dedicated mapping. The authors extend the unified segmentation model, a generative Gaussian mixture with a deformable tissue template, to handle voxels and entire channels that are missing. Once fitted to a population, the model can predict an entirely unacquired MR contrast from one or a few observed contrasts, or a CT image from MR scans, using the conditional Gaussian of the missing channels given the observed ones. The practical payoff is that translation inherits the benefits of a principled likelihood model: it works with small training sets, produces probabilistic predictions, and comes bundled with segmentation, bias correction, and spatial normalization.

What carries the argument

The workhorse is the unified segmentation model recast as a variational Gaussian mixture with Gauss-Wishart priors on each class's mean and precision, and a deformable template defining voxel-wise class probabilities. The key move is treating unobserved channels as missing data: the observed-channel likelihood marginalizes the Gaussian over missing dimensions (Eq. 6), and the expected marginal posterior over missing values, $E[h_d] = \sum_k \tilde{z}_{dk} \tilde{h}_{dk}$ from Eq. (9), is the synthetic image. Population parameters (template and prior hyperparameters) are learned by variational expectation-maximisation across training subjects, which is what makes the model empirical Bayesian.

What would settle it

Take a cohort with naturally truncated fields of view, infer the missing voxels with the trained model, and compare them against the same anatomy imaged in a separate full-coverage scan; if inferred intensities diverge systematically in exactly the regions where coverage is most often cut off, such as the posterior fossa, the missing-at-random assumption fails.

Watch

Extended reading notes

Core claim

The central claim is that if a group-wise generative model of brain scans — a variational Gaussian mixture with class labels driven by a deformable template — is allowed to treat unobserved channels as missing data, then the same fitted model that normalizes and segments a subject also outputs the missing images. For each voxel, the posterior over the missing contrasts is a mixture of Gaussians, and its expected value $E[h_d] = \sum_k \tilde{z}_{dk} \tilde{h}_{dk}$ gives the predicted intensities. The authors demonstrate this on three tasks: filling voxels absent because of differing fields of view, predicting entirely missing MR contrasts in all permutations of observed and missing channels, and synthesizing CT from T1w and T2w MR images. They report that the model produces realistic images, that T1w is the most predictive contrast, and that MR-to-CT translation from only eight training subjects reaches a mean PSNR of 25.5 dB.

Load-bearing premise

The whole prediction rests on the assumption that missingness is ignorable: whether a voxel's channel is unobserved must be unrelated to what its value would have been, which the paper never states or tests.

Editorial extensions

If this is right

  • Missing fields of view in routine clinical MR scans can be filled in rather than discarded, with mean PSNR above 36 dB when half the field of view is missing.
  • An entirely missing MR contrast can be predicted from any combination of remaining contrasts, with T1w the most informative; predicted contrasts are accurate enough to serve as inputs to deep-learning pipelines trained on a fixed contrast.
  • CT images can be synthesized from T1w and T2w MR scans after training on as few as eight subjects, offering a radiation-free route to attenuation correction in PET-MR.
  • The same fitted model yields segmentation, bias correction, and spatial normalization alongside translation, so downstream analysis needs no separate preprocessing.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same machinery could be pushed to scanner harmonization: fitting the joint model on multi-site data and generating a template-contrast image for each subject would isolate site-specific intensity shifts from biological variability.
  • The missing-at-random assumption, which the paper leaves implicit, can be tested directly by comparing synthetically masked voxels (where the assumption holds by construction) with naturally missing voxels from truncated acquisitions; systematic divergence would indicate bias in real-world applications.
  • The posterior variance of the expected marginal could serve as a per-voxel confidence map, flagging regions where the synthesized modality is unreliable — a guardrail that adversarial image translation methods lack.
  • A natural extension is to optimize the template on the target modality as well as the intensity hyperparameters; the paper notes that the meninges and bone detail are missed when only intensities are retrained, so joint template-intensity fitting could close that gap.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper presents a generative model for medical image translation based on an empirical Bayesian Gaussian mixture model. The unified segmentation model of Ashburner and Friston is extended to handle missing data, so that a multivariate Gaussian mixture with a deformable tissue template can be trained on partially observed multi-channel images and then used to predict missing modalities (MR contrasts or CT) for unseen subjects. The training procedure is a variational EM algorithm in which subject-specific posteriors are computed with a mean-field approximation and population prior hyper-parameters are estimated from the data. Experiments are reported on three tasks: inferring missing spatial voxels in MR volumes with different fields of view (IXI dataset), inferring entirely missing MR contrasts (IXI), and translating T1w/T2w MRI into CT (RIRE). All quantitative results are given as PSNR computed on held-out subjects. The paper's central claim is that image translation is a by-product of learning a joint intensity distribution, and that the model works with a small number of training subjects.

Significance. If the findings are supported, the proposed model would be a valuable interpretable alternative to deep-learning-based image translation, with the same model providing segmentation, spatial normalization, and missing-data synthesis in a single probabilistic framework. The derivation of the variational updates (Eqs. 12–23) is standard and internally consistent, and the use of held-out subjects for all reported PSNR values is a clear strength. The paper also honestly acknowledges several limitations, including the qualitative comparison with deep learning and the failure to capture detailed bone structure in CT synthesis. However, the quantitative evidence for the headline claim is weakened by the choice of evaluation metric (whole-volume PSNR without excluding background), the absence of baseline comparisons on the same data, and an unstated missingness assumption. These issues are addressable, but they need to be resolved before the central claim is fully convincing.

major comments (3)
  1. [Section 3, Eq. (24)] The paper explicitly states that "no voxels are excluded when computing the PSNR," and this is the only quantitative metric used. Head MR and CT volumes contain a large fraction of air/background voxels with near-constant intensities, so a Gaussian mixture component with a small-variance background class can score those voxels almost perfectly. This can substantially inflate the mean PSNR values reported in Tables 1 and 2 and the MR-to-CT result in Section 3.2. Please recompute PSNR within a brain mask (or a tissue mask) and report both whole-volume and masked values, so that the claim that the model "does a good job at filling in missing fields of view" is supported by tissue-only accuracy.
  2. [Section 3, Tables 1 and 2] No baseline comparison is provided on the same held-out data. The PSNR values are reported only for the proposed model, and the reference to prior work in Section 3.1 is a vague statement that results are "close to those previously reported" without citing specific numbers. Similarly, the CT experiment in Section 3.2 states the result is "not on pair with deep learning based techniques" but does not give any quantitative comparison. Please add at least one baseline on the same data splits (for example, a conditional-mean imputation from a complete-case Gaussian mixture, or a published synthesis method) so that the performance gain of the proposed model is demonstrated rather than assumed.
  3. [Section 2, Eqs. (6)–(7)] The derivation of the observed-channel likelihood and the conditional prediction of missing channels assumes that missingness is ignorable, i.e., the probability that a channel is missing does not depend on the missing intensity value. This assumption is never stated or tested in the manuscript. In clinical data, missingness may be associated with anatomy, pathology, or field-of-view choices, in which case the predicted values from Eq. (9) would be biased. Please state the missingness assumption explicitly and discuss its validity; a sensitivity analysis with simulated non-ignorable missingness would strengthen the claim that the model has clinical utility.
minor comments (6)
  1. [Eq. (2)] The equation for the categorical prior is not typeset clearly: the denominator is missing a separating space and the definition of a_dk(φ) is deferred to later text. Please revise for readability.
  2. [Eq. (17)] The subscripts o and m in the expectations E[z_dk x_d]_o, E[z_dk x_d]_m, etc., are not self-explanatory. Define these subscripts (observed and missing parts) before presenting the update equations.
  3. [Section 3.1] The text says the model is trained on "50 subjects from the IXI dataset" and then specifies "IXI[064-118]" and "IXI[002-063]"; please verify that these indices each correspond to 50 subjects and state the counts explicitly.
  4. [Section 3.1] The comparison with Chartsias et al. is only qualitative; please include the exact PSNR value from that reference when you say the results are "close", or remove the comparison.
  5. [Conclusions] There is a typo in "can then be drived to zero" (should be "driven"). Please correct this and any other grammatical errors.
  6. [Reproducibility] The paper does not mention code or data availability. For a methodological paper, please include a statement about whether the implementation will be released.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: held-out predictions are genuine forecasts of unseen modalities under a generative model.

full rationale

No circularity found. The derivation chain from Eqs. (6)-(9) is a standard conditional-Gaussian computation under a variational Gaussian mixture: Eq. (6) marginalizes observed channels, Eq. (7) gives the conditional distribution of missing channels, and Eq. (9) forms the posterior predictive mean. The estimator E[hd] = sum_k z~dk h~dk is evaluated on held-out subjects whose missing channels are not used to fit the population hyper-parameters or the subject-level posterior, so the reported PSNR values in Tables 1-2 and the RIRE experiment are genuine predictions rather than fitted inputs renamed as predictions. The paper's reliance on prior work for template update rules, e.g., 'We do not provide update rules for the template (a), as they can be found in [17]', is a normal citation of an established method and is not load-bearing for the central claim, which is demonstrated empirically on unseen subjects. Concerns about whole-volume PSNR including background voxels are a metric-strength issue, not evidence of circularity, and the missing-at-random assumption is an unstated correctness risk, not a circular dependency.

Assumptions & free parameters 2 free parameters · 6 assumptions · 0 invented entities

The central claim rests on standard probabilistic modeling assumptions: Gaussian mixtures, conjugate priors, mean-field variational inference, and a deformable atlas. There are no invented physical entities. The main hand-set free parameter is the number of mixture components K; the global class proportions omega are estimated as part of the model. The mixture means, covariances, and template are learned from training data under explicit assumptions, so they are not unacknowledged free parameters.

free parameters (2)
  • number of Gaussian mixture components K = 12
    Set by hand; the authors state that too low a K makes simulated images unrealistic and that K=12 was found empirically to capture a detailed enough distribution (Section 3.1 and Conclusion).
  • global class proportions omega = estimated, not reported
    Optimized as point estimates as part of the model to account for variable amounts of tissue classes, and noted as enabling automatic relevance determination (Eq. 2 and Conclusion).
assumptions (6)
  • domain assumption Missing channels are missing at random, so the observed likelihood is obtained by marginalizing the Gaussian over missing channels.
    The model never models why a channel is unobserved; if missingness correlates with intensity or tissue type, the predicted E[hd] is biased.
  • domain assumption Class-conditional intensities are multivariate Gaussian within each of K tissue classes.
    The conditional expectation that predicts missing values is Gaussian; non-Gaussian intensity distributions are only approximated.
  • domain assumption The mean-field factorization of the posterior is a good approximation.
    Exact marginalization is intractable, and the error introduced by the variational approximation is not quantified.
  • domain assumption A deformable softmax template with global class proportions describes the spatial distribution of classes.
    This is the standard unified-segmentation prior; it assumes a common atlas can be deformed to each subject.
  • domain assumption Conjugate Gauss-Wishart priors adequately capture population variability of intensity parameters.
    This modeling choice enables closed-form updates, but its adequacy for the data is not tested.
  • domain assumption The template learned from healthy IXI subjects transfers to RIRE patients with pathology, thick slices, and different scanners.
    In the MRI-to-CT experiment, the template is retained from IXI and only intensity hyperparameters are refit; the authors acknowledge this may limit bone and CSF modeling.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Empirical Bayesian Mixture Models for Medical Image Translation." pith.science (2026). https://pith.science/paper/LE4LGHDE

@misc{pith2026190805926,
  author       = {Pith},
  title        = {Pith review of: Empirical Bayesian Mixture Models for Medical Image Translation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/LE4LGHDE}},
  note         = {Machine review of arXiv:1908.05926}
}
read the original abstract

Automatically generating one medical imaging modality from another is known as medical image translation, and has numerous interesting applications. This paper presents an interpretable generative modelling approach to medical image translation. By allowing a common model for group-wise normalisation and segmentation of brain scans to handle missing data, the model allows for predicting entirely missing modalities from one, or a few, MR contrasts. Furthermore, the model can be trained on a fairly small number of subjects. The proposed model is validated on three clinically relevant scenarios. Results appear promising and show that a principled, probabilistic model of the relationship between multi-channel signal intensities can be used to infer missing modalities -- both MR contrasts and CT images.

Figures

Figures reproduced from arXiv: 1908.05926 by the authors.

Figure 1
Figure 1. Template and expectations of the Gaussians drawn from the Gauss [PITH_FULL_IMAGE:figures/full_fig_p007_1.png] view at source ↗
Figure 2
Figure 2. Example of inferring MRIs with differing field of views. An MR image [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 3
Figure 3. Example of inferring non-acquired MR contrasts. An MR image with two [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Example of MRI to CT translation. An MR image with two channels [PITH_FULL_IMAGE:figures/full_fig_p010_4.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. A Tool for Super-Resolving Multimodal Clinical MRI

    eess.IV 2019-09 conditional novelty 5.0 of 10

    A generative super-resolution model using multi-channel total variation reconstructs 1 mm isotropic images from thick-sliced multimodal clinical MRI, improving downstream age and sex prediction versus interpolation.

Reference graph

Works this paper leans on

23 extracted references · 22 canonical work pages · cited by 1 Pith paper

  1. [1]

    Attenuation correction synthesis for hybrid PET-MR scanners,

    N. Burgos, M. J. Cardoso, M. Modat, S. Pedemonte, J. Dickson, A. Barnes, J. S. Duncan, D. Atkinson, S. R. Arridge, B. F. Hutton, et al. , “Attenuation correction synthesis for hybrid PET-MR scanners,” in MICCAI, pp. 147–154, Springer, 2013

  2. [2]

    Regis- tration for correlative microscopy using image analogies,

    T. Cao, C. Zach, S. Modla, D. Powell, K. Czymmek, and M. Niethammer, “Regis- tration for correlative microscopy using image analogies,” in WBIR, pp. 296–306, Springer, 2012

  3. [3]

    MR contrast synthesis for lesion segmentation,

    S. Roy, A. Carass, N. Shiee, D. L. Pham, and J. L. Prince, “MR contrast synthesis for lesion segmentation,” in ISBI, pp. 932–935, IEEE, 2010

  4. [4]

    MRI modalitiy transformation in demon registra- tion,

    D.-J. Kroon and C. H. Slump, “MRI modalitiy transformation in demon registra- tion,” in ISBI, pp. 963–966, IEEE, 2009

  5. [5]

    Three-dimensional multi- modal brain warping using the demons algorithm and adaptive intensity correc- tions,

    A. Guimond, A. Roche, N. Ayache, and J. Meunier, “Three-dimensional multi- modal brain warping using the demons algorithm and adaptive intensity correc- tions,” IEEE T. Med. Imaging , vol. 20, no. 1, pp. 58–69, 2001

  6. [6]

    Automatic CT-ultrasound registration for diagnostic imaging and image-guided intervention,

    W. Wein, S. Brunke, A. Khamene, M. R. Callstrom, and N. Navab, “Automatic CT-ultrasound registration for diagnostic imaging and image-guided intervention,” Med. Image Anal. , vol. 12, no. 5, pp. 577–585, 2008

  7. [7]

    Investigation of a method for generating synthetic CT models from MRI scans of the head and neck for radiation therapy,

    S.-H. Hsu, Y. Cao, K. Huang, M. Feng, and J. M. Balter, “Investigation of a method for generating synthetic CT models from MRI scans of the head and neck for radiation therapy,” Phys. Med. Biol. , vol. 58, no. 23, p. 8419, 2013

  8. [8]

    Estimating CT image from MRI data using structured random forest and auto-context model,

    T. Huynh, Y. Gao, J. Kang, L. Wang, P. Zhang, J. Lian, and D. Shen, “Estimating CT image from MRI data using structured random forest and auto-context model,” IEEE T. Med. Imaging , vol. 35, no. 1, pp. 174–183, 2015

Show all 23 references
  1. [9]

    Is synthesizing MRI contrast useful for inter-modality analysis?,

    J. E. Iglesias, E. Konukoglu, D. Zikic, B. Glocker, K. Van Leemput, and B. Fischl, “Is synthesizing MRI contrast useful for inter-modality analysis?,” in MICCAI, pp. 631–638, Springer, 2013

  2. [10]

    A compressed sensing approach for MR tissue contrast synthesis,

    S. Roy, A. Carass, and J. Prince, “A compressed sensing approach for MR tissue contrast synthesis,” in IPMI, pp. 371–383, Springer, 2011

  3. [11]

    Multimodal mr syn- thesis via modality-invariant latent representation,

    A. Chartsias, T. Joyce, M. V. Giuffrida, and S. A. Tsaftaris, “Multimodal mr syn- thesis via modality-invariant latent representation,” IEEE transactions on medical imaging, vol. 37, no. 3, pp. 803–814, 2017

  4. [12]

    Medical image synthesis with context-aware generative adversarial networks,

    D. Nie, R. Trullo, J. Lian, C. Petitjean, and M. Ruan, “Medical image synthesis with context-aware generative adversarial networks,” pp. 417–425, Springer, 2017

  5. [13]

    Deep MR to CT synthesis using unpaired data,

    J. M. Wolterink, A. M. Dinkla, M. H. Savenije, P. R. Seevinck, C. A. van den Berg, and I. Iˇ sgum, “Deep MR to CT synthesis using unpaired data,” inSASHIMI, pp. 14–23, Springer, 2017

  6. [14]

    Distribution matching losses can hallucinate features in medical image translation,

    J. P. Cohen, M. Luck, and S. Honari, “Distribution matching losses can hallucinate features in medical image translation,” in MICCAI, pp. 529–536, Springer, 2018

  7. [15]

    Supervised learning from incomplete data via an EM approach,

    Z. Ghahramani and M. I. Jordan, “Supervised learning from incomplete data via an EM approach,” in NeurIPS, pp. 120–127, 1994

  8. [16]

    Unified segmentation,

    J. Ashburner and K. J. Friston, “Unified segmentation,” NeuroImage, vol. 26, no. 3, pp. 839–851, 2005

  9. [17]

    Generative diffeomor- phic modelling of large MRI data sets for probabilistic template construction,

    C. Blaiotta, P. Freund, M. J. Cardoso, and J. Ashburner, “Generative diffeomor- phic modelling of large MRI data sets for probabilistic template construction,” NeuroImage, vol. 166, pp. 117–134, 2018

  10. [18]

    Empirical Bayes: Past, present and future,

    B. P. Carlin and T. A. Louis, “Empirical Bayes: Past, present and future,” J. Am. Stat. Assoc., vol. 95, no. 452, pp. 1286–1289, 2000

  11. [19]

    Variational inference for medical image segmentation,

    C. Blaiotta, M. J. Cardoso, and J. Ashburner, “Variational inference for medical image segmentation,” Comput. Vis. Image Und. , vol. 151, pp. 14–28, 2016

  12. [20]

    C. M. Bishop, Pattern recognition and machine learning . Springer, 2006

  13. [21]

    Nonlinear markov random fields learned via backpropagation,

    M. Brudfors, Y. Balbastre, and J. Ashburner, “Nonlinear markov random fields learned via backpropagation,” in IPMI, pp. 805–817, Springer, 2019

  14. [22]

    Comparison and evaluation of retrospective intermodality image registration techniques,

    J. B. West, . . ., and R. P. Woods, “Comparison and evaluation of retrospective intermodality image registration techniques,” in Medical Imaging 1996: Image Pro- cessing, vol. 2710, pp. 332–348, SPIE, 1996

  15. [23]

    MRI super-resolution using multi-channel total variation,

    M. Brudfors, Y. Balbastre, P. Nachev, and J. Ashburner, “MRI super-resolution using multi-channel total variation,” in MIUA, pp. 217–228, Springer, 2018

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.