Pith. sign in

REVIEW 3 major objections 7 minor 9 references

Reliable Evaluation of MRI Motion Correction: Dataset and Insights

T0 review · 3 major / 7 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Real-world paired evaluation with the new PMoC3D dataset and MoMRISim metric reliably tracks human judgment of MRI motion correction, while simulated-motion and reference-free evaluation mislead.

desk verdict The raw k-space dataset is the real contribution; the reliability claim leans on a single human rater and needs stronger validation. read the letter →

arxiv 2506.05975 v1 pith:2LBGJAQP submitted 2025-06-06 eess.IV

Kun Wang , Tobit Klug , Stefan Ruschke , Jan S. Kirschke , Reinhard Heckel This is my paper
classification eess.IV
keywords MRImotioncorrectionPMoC3DdatasetMoMRISimevaluationmethodologyperceptualimagequalitymetricssimulatedartifactsreference-free3Dbrain
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Correcting motion artifacts in 3D brain MRI is clinically valuable, but the field has no agreed way to tell whether a correction method actually works, because true ground-truth scans of moving patients can never be obtained. This paper compares the three evaluation strategies in use—scoring against a motion-free reference scan, scoring against simulated motion, and scoring without any reference—and finds that two of them mislead. Simulated motion makes correction methods look dramatically better than they perform on real scans, even when artifact severity is matched; reference-free metrics, including a GPT-4o-based score, systematically overrate smooth deep-learning outputs that have lost anatomical detail. The paper's positive contribution is a real-world paired dataset, PMoC3D, together with a learned feature-space metric, MoMRISim, whose rankings agree with human perception (Spearman's $\rho = 0.95$) for moderate-to-severe motion. If this holds, the field finally has a benchmark and metric that measure real correction performance rather than simulation or smoothness.

What carries the argument

The argument is carried by three objects. PMoC3D is a dataset of unprocessed, paired 3D brain MRI acquisitions—one motion-free reference and three motion-corrupted scans per subject, with k-space (raw spatial-frequency) data, sensitivity maps, and per-shot motion instructions—so that motion-estimation methods that need raw data can be tested on real artifacts. MoMRISim is a learned perceptual metric: a DINO-vitb16 visual encoder fine-tuned with LoRA (a lightweight low-rank adapter) on triplets (reference, reconstruction A, reconstruction B) labeled by simulated motion severity, trained so that it ranks the less-corrupted reconstruction as closer to the reference; it thereby learns motion-artifact features without human annotation. PMAS, the perceived motion artifact score, is the yardstick of reliability: pairwise human comparisons are fit with a Bradley-Terry model, and every evaluation approach and metric is judged by how well its rankings agree with PMAS. The third ingredient is the three-way comparison protocol itself—real-world paired, simulated, and reference-free evaluation run on the same three baseline methods (alternating optimization, MotionTTT, and stacked U-nets)—which is what lets the paper attribute over- and under-rating to the evaluation approach rather than to any single algorithm.

What would settle it

Have several independent radiologists rank the same set of PMoC3D reconstructions pairwise, average their preferences into a consensus PMAS, and recompute correlations: if MoMRISim's Spearman correlation with the consensus drops well below 0.95, or if another metric matches or exceeds it, the paper's claim that MoMRISim is the most reliable reference-based metric would be falsified. A second check: retrain MoMRISim on a different brain dataset with the same triplet scheme and see whether its correlation with human judgment on PMoC3D is preserved, which would test whether the result depends on the specific training data.

Watch

Extended reading notes

Core claim

The central claim is that evaluation on real-world paired data—the PMoC3D benchmark the paper releases, scored with the MoMRISim metric it also introduces—gives a relatively reliable and meaningful measure of 3D MRI motion-correction performance under moderate to severe motion, and that the two popular alternatives do not. Using perceived motion artifact scores (PMAS), obtained from pairwise human comparisons fitted with a Bradley-Terry model, as the yardstick, the paper reports that MoMRISim's ranking of reconstructions correlates with human judgment at Spearman $\rho = 0.95$, above PSNR, SSIM, DISTS, DreamSim, and the proposed VLM score. On simulated motion, a blind comparison showed that in 75% of matched-severity pairs the real-world reconstruction had clearly more residual artifacts than its simulated counterpart, so simulation-based evaluation systematically exaggerates algorithm performance. Reference-free metrics, including the vision-language-model score proposed here, weakly align with human judgment and hand high scores to oversmoothed stacked U-net outputs that visibly lose anatomy. The paper also documents a boundary of its own recommendation: under mild motion, corrected images can look cleaner than the motion-free reference, so reference-based evaluation loses validity exactly where artifacts are already subtle.

Load-bearing premise

Reliability is defined as agreement with the perceived motion artifact score, but the pairwise human comparisons behind that score were made by a single evaluator (Appendix B), so the reported 0.95 correlation shows agreement with one person's judgment rather than with a consensus of radiologists.

Editorial extensions

If this is right

  • Papers evaluating 3D MRI motion correction should report results on real paired data like PMoC3D with a feature-based metric, since simulated-motion numbers will systematically overstate progress.
  • Simulated-motion evaluation remains useful only for relative comparisons or the mild-motion regime; matched-severity real artifacts are consistently harder to correct.
  • Reference-free metrics, including VLM-based scores, are not reliable for ranking deep-learning motion-correction methods because they reward smoothness at the cost of anatomy.
  • Motion-correction methods that estimate motion parameters from raw k-space can finally be evaluated on real motion, because PMoC3D provides unprocessed measurements rather than only magnitude images.
  • For mild motion, reference-based scoring is unsafe: a corrected image can legitimately beat the motion-free reference, so benchmark design must separate mild from moderate and severe cases.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A multi-rater study of the same reconstructions would test how stable the reported 0.95 correlation is; if agreement between raters is low, the metric ranking itself may change.
  • MoMRISim is trained on simulated motion yet validated against real motion—if that transfer holds, the same triplet scheme could be extended to other artifact families (non-rigid motion, pulsation, spin-history effects) that current simulation cannot capture.
  • The VLM-score failure is a caution for the wider practice of using vision-language models as automatic image-quality judges: without explicit anti-oversmoothing constraints, such scores will systematically favor plausible-looking but detail-poor reconstructions.
  • Because PMoC3D records per-shot motion instructions and time stamps, it could double as a testbed for predicting motion severity from raw data, not just for evaluating correction algorithms.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 7 minor

Summary. The paper addresses the evaluation of 3D MRI motion correction methods. It introduces PMoC3D, a dataset of paired raw k-space motion-corrupted and motion-free scans from 8 subjects, and MoMRISim, a learned feature-space similarity metric trained on triplets labeled by simulated motion severity. The authors compare three evaluation strategies: real-world reference-based evaluation, simulated-motion evaluation, and reference-free evaluation. Their main findings are that (i) real-world evaluation combined with MoMRISim correlates well with human-perceived artifact scores for moderate-to-severe motion, (ii) simulated-motion evaluation systematically overestimates algorithm performance, and (iii) reference-free metrics, including a proposed VLM score, align poorly with human judgment and can favor oversmoothed outputs. The dataset and code are released.

Significance. The PMoC3D dataset release is a valuable contribution: raw paired k-space data with motion corruption is scarce, and the paper provides a thoughtful discussion of the mild-motion failure mode of reference-based evaluation. If the central claims hold, the paper would establish a preferred benchmark and metric for 3D MRI motion correction evaluation. The validation of MoMRISim against an independent human-based score is a strength, as is the honest reporting of limitations. However, the empirical support is weakened by the small sample size (8 subjects), the use of a single-annotator ground truth for the key correlation, and the absence of confidence intervals or inter-rater agreement measures. The claims should therefore be treated as preliminary until the authors provide uncertainty quantification and ideally multi-rater evidence.

major comments (3)
  1. [Section 4.2 and Appendix B] The central claim that MoMRISim provides a reliable real-world evaluation metric rests on the Spearman correlation rho=0.95 with the perceived motion artifact score (PMAS). Appendix B states that for the PMAS used in Section 4.2, 'annotations were performed by a single evaluator.' No inter-rater agreement, rater blinding, or confidence interval for the correlation is reported. Because the central claim defines reliability as agreement with human perception, this single-annotator ground truth is load-bearing. I recommend adding a second (or more) rater on at least a subset of the comparisons, reporting inter-rater agreement, and providing bootstrap confidence intervals on the reported correlations. Alternatively, the claims should be explicitly limited to agreement with this one annotator.
  2. [Section 4.3] The conclusion that simulated-motion evaluation systematically exaggerates algorithm performance is primarily supported by a 64-pair comparison performed by a single 'human annotator,' with no reported expertise, blinding procedure, or inter-rater reliability. The matching of artifact severity between real-world and simulated volumes is described only qualitatively ('closely resembling'), which could bias the comparison if the simulated volumes were in fact more severely corrupted. Please provide the exact number of simulated volumes, the severity-matching procedure (including any quantitative criteria), and ideally a second annotator or a sensitivity analysis.
  3. [Section 4.2 and Appendix F.1] The correlations in Figure 2 and Appendix F.1 are reported without confidence intervals or significance tests on the differences between metrics (e.g., MoMRISim rho=0.95 versus DreamSim rho=0.93). The sample arises from 8 subjects, and the reconstruction-level PMAS involves only 48 or 72 volumes; with such a small sample, the ordering of metrics may not be stable. Please provide bootstrap confidence intervals on all reported Spearman correlations and test whether the difference between MoMRISim and DreamSim is statistically significant.
minor comments (7)
  1. [Figure 2] The x-axis extends to 6 while the PMAS values in Table 2 lie in [-0.44, 2.42]; the caption also says 'medium and severe' but negative PMAS values (mild) appear. Please clarify the axis range and the subset of scans actually shown.
  2. [Section 3.2] An acceleration factor of 4.9 is reported, whereas Section 2.1 and Table 1 report 4.94; please make these values consistent.
  3. [Appendix C.1 and C.2] The learning rates and regularization weights contain missing superscripts ('learning rate of 10 8' should be 10^{-8}; 'lambda = 3-8' should be 3 x 10^{-8}).
  4. [Figure 2 caption] The caption reads 'MoMRIScore' instead of 'MoMRISim'; please correct the typo.
  5. [Section 2.1] The sentence 'We provide access to the corrupted scans (S3 4, S5 4, S8 4) with involuntary motion resulting in a total of 27 motion-corrupted scans' is unclear: the notation and the total count (24 voluntary plus 3 involuntary?) should be explained.
  6. [Appendix D] The reported triplet-ranking accuracy of 0.933 is computed on the training set; please also report validation accuracy to indicate generalization.
  7. [Appendix E] The VLM score uses GPT-4o at temperature 0.5, but no specific model version or access date is given, which limits reproducibility; please add this information.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: MoMRISim is validated against independent human PMAS on real-world data, and the self-citations to the authors' simulation protocol are not load-bearing.

full rationale

MoMRISim is trained on synthetic triplets whose labels are simulated motion severity levels (Section 3.4; Appendix D) and is then validated against PMAS, which is fitted with a Bradley-Terry model on human pairwise comparisons of real PMoC3D reconstructions (Section 4.2; Appendix B). These label sources are independent: the simulated severity labels come from the authors' [Klu+24] protocol, while PMAS comes from human judgments of real reconstructions; neither is computed from the other. The self-citation to [Klu+24] supplies the synthetic motion model and the MotionTTT baseline, but the paper's central claims do not reduce to it: the rho=0.95 reliability claim is measured against human PMAS, and the simulated-versus-real overestimation finding is assessed by a blind human comparison (Section 4.3). The reconstruction-level PMAS being produced by a single evaluator and the small number of subjects are threats to the external validity of the human ground truth, not definitional reductions; they are correctness and statistical risks rather than circularity. Similarly, the VLM prompt explicitly includes oversmoothing checks, so the finding that such a metric still overrates stacked U-net outputs is an empirical failure of reference-free evaluation, not a result built into the metric's definition. No equation or fitted parameter is reused as a prediction, and no load-bearing uniqueness claim is imported from the authors' prior work.

Assumptions & free parameters 2 free parameters · 5 assumptions · 0 invented entities

The central claim rests primarily on the PMAS ground truth from a single annotator and on transfer from simulated training data. The metric itself is a learned network rather than a scalar fitted to the target data, so no single free parameter is load-bearing beyond the chosen simulation severity levels and baseline hyperparameters.

free parameters (2)
  • Simulation severity levels for MoMRISim training and simulated evaluation = Mild: 1 event, up to ±5 degrees/mm; Severe: 3 events, up to ±15 degrees/mm
    Hand-chosen by the authors to define motion severity for training the metric and for simulated evaluation; the central reliability conclusion does not depend on these exact values but the metric's behavior could.
  • MotionTTT DC loss threshold = 0.70
    Hyperparameter used to reject corrupted shots before final reconstruction; influences baseline reconstruction quality and thus the comparison, but is not central to the evaluation-methodology claim.
assumptions (5)
  • domain assumption The Calgary Campinas dataset (CC-359) is representative of T1-weighted brain MRI for training MoMRISim and baselines.
    Used to train MoMRISim and the deep learning baselines; if this dataset is not representative of PMoC3D, the metric and methods may transfer poorly.
  • domain assumption Simulated rigid inter-shot motion with event-based random perturbations is a useful proxy for real motion when training the metric.
    MoMRISim is trained on synthetic motion severity labels; the paper argues this transfers by showing correlation with real-world PMAS, but the assumption is built into the method.
  • domain assumption The L1-wavelet reconstruction of the motion-free scan is an adequate reference for real-world evaluation.
    The paper uses this as ground truth for computing reference-based metrics; it acknowledges residual artifacts in the reference, especially under mild motion.
  • domain assumption A single annotator's pairwise comparisons are a valid measure of perceived motion artifact severity (PMAS).
    The reliability of real-world evaluation is measured by correlation with this PMAS; no inter-rater agreement is reported.
  • standard math Bradley-Terry model converts pairwise preference data into a valid interval score.
    Standard statistical model for pairwise comparisons; used to derive PMAS.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Reliable Evaluation of MRI Motion Correction: Dataset and Insights." pith.science (2026). https://pith.science/paper/2LBGJAQP

@misc{pith2026250605975,
  author       = {Pith},
  title        = {Pith review of: Reliable Evaluation of MRI Motion Correction: Dataset and Insights},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2LBGJAQP}},
  note         = {Machine review of arXiv:2506.05975}
}
read the original abstract

Correcting motion artifacts in MRI is important, as they can hinder accurate diagnosis. However, evaluating deep learning-based and classical motion correction methods remains fundamentally difficult due to the lack of accessible ground-truth target data. To address this challenge, we study three evaluation approaches: real-world evaluation based on reference scans, simulated motion, and reference-free evaluation, each with its merits and shortcomings. To enable evaluation with real-world motion artifacts, we release PMoC3D, a dataset consisting of unprocessed Paired Motion-Corrupted 3D brain MRI data. To advance evaluation quality, we introduce MoMRISim, a feature-space metric trained for evaluating motion reconstructions. We assess each evaluation approach and find real-world evaluation together with MoMRISim, while not perfect, to be most reliable. Evaluation based on simulated motion systematically exaggerates algorithm performance, and reference-free evaluation overrates oversmoothed deep learning outputs.

Figures

Figures reproduced from arXiv: 2506.05975 by the authors.

Figure 1
Figure 1. Sagittal views and corresponding perceived motion artifact scores (PMAS) of selected [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Correlation plot of PSNR↑, MoMRISim↓, and VLM score↓ with the perceived motion artifact score. MoMRIScore shows a strong correlation, PSNR offers a moderate level, and VLM score reflects a low degree of alignment. MotionTTT Recon. Reference Difference [PITH_FULL_IMAGE:figures/full_fig_p009_2.png] view at source ↗
Figure 3
Figure 3. Comparison of MotionTTT reconstruction, L1-based motion-free reference, and their [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figures from the paper (9 more)
Figure 4
Figure 4. Figure 4: Baseline reconstructions of scans corrupted with simulated and PMoC3D artifacts and [PITH_FULL_IMAGE:figures/full_fig_p011_4.png]
Figure 5
Figure 5. Figure 5: Panel a): schematic visualization of the magnitude of a 3D volume; Panel b): the [PITH_FULL_IMAGE:figures/full_fig_p016_5.png]
Figure 6
Figure 6. Figure 6: Example triplet from the MoMRISim training dataset. The left image is corrupted by [PITH_FULL_IMAGE:figures/full_fig_p019_6.png]
Figure 7
Figure 7. Figure 7: An example input for VLM evaluation, where three slices from sagittal, coronal, and axial [PITH_FULL_IMAGE:figures/full_fig_p022_7.png]
Figure 8
Figure 8. Figure 8: Correlation plot of PSNR↑, SSIM↑, and AP↓ with the perceived motion artifact score. All of them show moderate correlation with human judgment. −4 −2 0 2 4 6 PMAS 0.04 0.06 0.08 0.10 0.12 0.14 0.16 0.18 Spearman ρ =0.85 DISTS −4 −2 0 2 4 6 PMAS 0.02 0.04 0.06 0.08 0.10 …
Figure 9
Figure 9. Figure 9: Correlation plot of DISTS↓, DreamSim↓, and MoMRISim↓ with the perceived motion artifact score. All of them show high correlation with human judgment, while the MoMRISim shows the highest correlation. −4 −2 0 2 4 PMAS 0.12 0.14 0.16 0.18 0.20 0.22 0.24 0.26 Spearman ρ =…
Figure 10
Figure 10. Figure 10: Correlation plot of TG↑, AES↑, and VLM Score↓ with the perceived motion artifact score. All of them show poor correlation with human judgment. 23 [PITH_FULL_IMAGE:figures/full_fig_p023_10.png]
Figure 11
Figure 11. Figure 11: Reconstructions from the PMoC3D (real) and simulated datasets under two reconstruction [PITH_FULL_IMAGE:figures/full_fig_p025_11.png]
Figure 12
Figure 12. Figure 12: Baseline reconstructions of scan S4 2, with the difference images and the calculated reference-free scores. 26 [PITH_FULL_IMAGE:figures/full_fig_p026_12.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

9 extracted references · 9 canonical work pages

  1. [1]

    Using deep feature distances for evaluating the perceptual quality of MR image reconstructions

    [Ada+25] P. M. Adamson et al. “Using deep feature distances for evaluating the perceptual quality of MR image reconstructions”. In: Magnetic Resonance in Medicine (2025). [And+15] J. B. Andre, B. W. Bresnahan, M. Mossa-Basha, M. N. Hoff, C. P. Smith, Y. Anzai, and W. A. Cohen. “Toward Quantifying the Prevalence, Severity, and Cost Associated With Patient ...

    work page 2025

  2. [6]

    Compressed Sensing MRI

    [LLC] G. LLC. ReconFrame. https : / / www . gyrotools . com / gt / index . php / products / reconframe. Software, accessed 2025-02-26. Zurich, Switzerland. [Lus+08] M. Lustig, D. L. Donoho, J. M. Santos, and J. M. Pauly. “Compressed Sensing MRI”. In: IEEE Signal Processing Magazine (2008). [Mar+24] E. Marchetto, H. Eichhorn, D. Gallichan, J. A. Schnabel, ...

    work page 2008

  3. [7]

    Quantitative framework for prospective motion correction evaluation

    [Pan+16] N. A. Pannetier, T. Stavrinos, P. Ng, M. Herbst, M. Zaitsev, K. Young, G. Matson, and N. Schuff. “Quantitative framework for prospective motion correction evaluation”. In: Magnetic Resonance in Medicine (2016). [RFB15] O. Ronneberger, P. Fischer, and T. Brox. “U-net: Convolutional networks for biomedical image segmentation”. In: Medical Image Com...

    work page 2016

  4. [9]

    Joint Frequency and Image Space Learning for MRI Reconstruction and Analysis

    [Sin+22] N. M. Singh, J. E. Iglesias, E. Adalsteinsson, A. V. Dalca, and P. Golland. “Joint Frequency and Image Space Learning for MRI Reconstruction and Analysis”. In: The Journal of Machine Learning for Biomedical Imaging (2022). [Sli+20] J. M. Slipsager et al. “Quantifying the Financial Savings of Motion Correction in Brain MRI: A Model-Based Estimate ...

    work page 2022

  5. [2010]

    PULSAR: A Matlab Toolbox for Parallel Magnetic Resonance Imaging Using Array Coils and Multiple Channel Receivers

    12 [JSR07] J. X. Ji, J. B. Son, and S. D. Rane. “PULSAR: A Matlab Toolbox for Parallel Magnetic Resonance Imaging Using Array Coils and Multiple Channel Receivers”. In: Concepts in Magnetic Resonance Part B: Magnetic Resonance Engineering (2007). [JD19] P. M. Johnson and M. Drangova. “Conditional Generative Adversarial Network for 3D Rigid-Body Motion Cor...

    work page 2007

  6. [2015]

    Implicit data crimes: Machine learning bias arising from misuse of public data

    [Shi+22] E. Shimron, J. I. Tamir, K. Wang, and M. Lustig. “Implicit data crimes: Machine learning bias arising from misuse of public data”. In: Proceedings of the National Academy of Sciences (2022). 13 [Sin+23] N. M. Singh, N. Dey, M. Hoffmann, B. Fischl, E. Adalsteinsson, R. Frost, A. V. Dalca, and P. Golland. “Data Consistent Deep Rigid MRI Motion Corr...

    work page 2022

  7. [2021]

    Sensitivity Encoding for Aligned Multishot Magnetic Resonance Recon- struction

    [Cor+16] L. Cordero-Grande, R. P. A. G. Teixeira, E. J. Hughes, J. Hutter, A. N. Price, and J. V. Hajnal. “Sensitivity Encoding for Aligned Multishot Magnetic Resonance Recon- struction”. In: IEEE Transactions on Computational Imaging (2016). [Din+22] K. Ding, K. Ma, S. Wang, and E. P. Simoncelli. “Image Quality Assessment: Unifying Structure and Texture ...

    work page 2016

  8. [2023]

    Network Accelerated Motion Estimation and Reduction (NAMER): Convolutional Neural Network Guided Retrospective Motion Correction Using a Separable Motion Model

    [GE22] M. Ganz and H. Eichhorn. “Datasets with and without Deliberate Head Movements for Evaluating the Performance of Markerless Prospective Motion Correction and Selective Reacquisition in a General Clinical Protocol for Brain MRI”. In: OpenNeuro (2022). [Has+19] M. W. Haskell, S. F. Cauley, B. Bilgic, J. Hossbach, D. N. Splitthoff, J. Pfeuffer, K. Sets...

    work page 2022

Show all 9 references
  1. [2024]

    PyWavelets: A Python Package for Wavelet Analysis

    [Lee+19] G. R. Lee, R. Gommers, F. Waselewski, K. Wohlfahrt, and A. O’Leary. “PyWavelets: A Python Package for Wavelet Analysis”. In: Journal of Open Source Software (2019). [LJT23] B. Levac, A. Jalal, and J. I. Tamir. “Accelerated Motion Correction for MRI Using Score-Based G...

    2019

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.