REVIEW 4 major objections 3 minor 31 references
All-in-One Medical Image Restoration with Latent Diffusion-Enhanced Vector-Quantized Codebook Prior
T0 review · 4 major / 3 minor · reviewed 2026-08-15 · deepseek-v4-flash
Pith's one-line read DiffCode proposes a task-adaptive codebook bank plus latent diffusion that restores MRI, CT, and PET images with one model and reports the best average scores among twelve methods.
desk verdict A plausible codebook-prior architecture for all-in-one MedIR, undermined by weak evidence for its signature latent-diffusion component. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the task-adaptive codebook bank: a set of vector-quantized codebooks, one per task, each containing 8,192 code items of dimension 256 and updated by residual vector quantization across eight iterations. The latent diffusion stage, built from a five-layer denoising network run for eight time steps, maps degraded-image-conditioned Gaussian noise back toward high-quality latent features before those features are quantized against the bank. A task-aware global routing module with four experts, one activated per image, routes each task's features through specialized subnetworks to reduce task interference.
What would settle it
On a held-out split, extract the true high-quality features for each test image, quantize them against the codebook bank, and compare the retrieved codes or feature distances from the diffusion path versus a one-step direct prediction from the degraded features. If the diffusion path does not improve retrieval accuracy or feature similarity, the paper's explanation for its gains is unsupported.
Extended reading notes
Core claim
The paper proposes that the obstacle to all-in-one medical image restoration is not the restoration backbone but the retrieval of high-quality prior information: each task destroys different features, so a single shared prior cannot compensate for all losses. DiffCode therefore builds a codebook bank in which each task has its own residual-quantized codebook of high-quality latent features, and then trains a latent diffusion model that, conditioned on the degraded image's features, iteratively refines a noisy latent toward the correct high-quality feature before quantization. The retrieved codes are decoded into a reference image that is concatenated with the degraded input and fed to a restoration network with task-aware routing. The paper reports that this pipeline outperforms eleven task-specific, general, and all-in-one baselines on MRI super-resolution, CT denoising, and PET synthesis, and that each component—codebook bank, latent diffusion, and routing—contributes positively in the ablations.
Load-bearing premise
The load-bearing premise is that a five-layer diffusion network can recover the correct high-quality latent feature from Gaussian noise in only eight denoising steps, conditioned on the degraded image; the paper validates this only through final restoration metrics, not by checking the retrieved features themselves.
Editorial extensions
If this is right
- One trained model can serve MRI super-resolution, CT denoising, and PET synthesis without task-specific fine-tuning, simplifying clinical deployment.
- Adding a new task may only require appending a new codebook and retraining the retrieval stage, rather than redesigning the whole restoration network.
- The codebook prior supplies high-quality reference images that could be used for downstream quality control or as inputs to other analysis models.
- Because the codebook bank and diffusion retriever are built once and then frozen, they can guide any restoration backbone, not only the NAF-block architecture used here.
- The reported gains over AMIR, the closest all-in-one baseline, suggest that the bottleneck in all-in-one restoration is prior retrieval rather than routing alone.
Reading between the lines
- A testable implication the paper leaves implicit is whether the diffusion retriever's benefit grows with degradation severity; if retrieval errors increase as blur or noise worsens, the fixed eight-step schedule may need task-dependent length.
- The codebook bank could in principle be extended to unseen tasks by adding codebooks trained on small amounts of high-quality data, turning restoration into a plug-and-play prior library; the paper does not demonstrate this.
- Comparing the retrieved code indices against ground-truth indices would isolate where the diffusion stage helps, since final PSNR conflates retrieval accuracy with the backbone's ability to ignore bad codes.
- The same latent-diffusion-plus-codebook pattern may transfer to non-medical multi-degradation restoration, where task heterogeneity is equally pronounced; this is an extrapolation, not a claim in the paper.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes DiffCode, a three-stage framework for all-in-one medical image restoration. Stage I builds a task-adaptive vector-quantized codebook bank with residual quantization on high-quality images for each task. Stage II trains a small latent diffusion model, conditioned on low-quality features, to estimate features that are then quantized against the codebook bank; the stated goal is to improve prior retrieval by refining the latent feature distribution. Stage III decodes the retrieved codebook features into reference images and concatenates them with the low-quality input to guide a task-aware routing restoration backbone. The authors report state-of-the-art results on MRI super-resolution, CT denoising, and PET synthesis, with average PSNR 34.6186, SSIM 0.9336, and RMSE 12.3728, and they provide component ablations in Table 2.
Significance. If the results are reproducible and the diffusion mechanism is genuinely improving prior retrieval, the task-adaptive codebook bank is a sensible and potentially useful contribution to all-in-one medical image restoration, where task heterogeneity is a recognized challenge. The paper is clearly written, the architecture is specified in detail, and the experimental comparison against eleven methods across three tasks is comprehensive. The component ablation shows that each module adds some quantitative benefit. However, the central novelty of the paper is the latent diffusion stage, and that stage is currently supported only by a small end-task metric gain with no direct evidence about retrieval behavior or distribution alignment; this weakens the significance of the claimed mechanism.
major comments (4)
- [Sec. 2.2 and Table 2(a)] The central claim that the latent diffusion model 'enhances prior retrieval' is supported only by a 0.0913 dB average PSNR improvement (V3: 34.5273 vs. DiffCode: 34.6186) and a 0.0013 SSIM improvement, reported without error bars or significance tests. No direct evidence is provided that the diffusion process changes which codebook entries are retrieved or that the estimated features align with ground-truth HQ features. This is load-bearing because the diffusion stage is the paper's main novelty; I request direct retrieval analysis (e.g., retrieved code index agreement with the HQ code indices, feature-space distance or MMD before vs. after diffusion) and repeated-seed statistics to rule out that the gain comes from added stochasticity or parameter count rather than from improved retrieval.
- [Sec. 2.2, Eq. (5)] At inference the reverse diffusion is initialized from randomly sampled Gaussian noise, but the experimental tables report a single run with no seed or variance information. Given that the largest reported advantage over the strongest baseline AMIR is 0.2601 dB average PSNR (Table 1), run-to-run variability could plausibly change the ranking. The authors should report mean and standard deviation over at least three to five seeds for all main comparisons and ablations, or fix a seed and state it explicitly.
- [Sec. 2.2, Eq. (6)] The training objective L_stage2 = ||z^n - \hat z^n||_1 trains the denoiser to map Gaussian noise to HQ features conditioned on LQ features, and at inference the reverse process starts from pure noise. This is not a 'refinement' of the LQ-degraded distribution in the sense of gradually correcting an input feature; it is a conditional generation from noise. The paper does not explain why this procedure should land near the codebook manifold, and no codebook reconstruction loss or nearest-neighbor constraint is used in Stage II. I ask the authors to show that \hat z^n lies close to the codebook manifold (e.g., nearest-neighbor distances before and after diffusion) and to ablate the initialization choice (Gaussian noise vs. LQ feature plus noise).
- [Sec. 3.1, PET Synthesis dataset] The description 'subsampling full scans with a 12x dose reduction, and reconstructed using the standard OSEM method' omits essential details: how the sinogram subsampling is performed, the OSEM iteration and subset counts, whether any post-filtering is applied, and how the high-quality reference is exactly defined. Without these details the PET synthesis result cannot be reproduced or compared fairly against the cited prior work [9], and the quantitative gain on that task is not fully verifiable.
minor comments (3)
- [Sec. 1, Contribution 1] The claim of being 'the first to exploit the VQ codebook prior for all-in-one MedIR' should be supported by a brief discussion of prior VQ-based restoration works and why none of them addresses the all-in-one setting; otherwise the novelty claim is too strong.
- [Fig. 1] Several labels in the overview figure appear as unreadable '???' placeholders, particularly around the equations and block annotations; please provide a clean vector version so that the architecture can be verified.
- [Sec. 3.1 and References] The implementation description gives NAF block counts and channels but omits the exact structure of the condition encoder E_lq, the number of learnable parameters, and the total compute or storage overhead of the codebook bank and diffusion stage; adding a parameter/FLOPs table would strengthen the efficiency claims in the Introduction. Also, reference [18] is missing its publication venue (ICLR 2017).
Circularity Check
No significant circularity: the codebook and latent diffusion stages are supervised training with independent held-out evaluation.
full rationale
DiffCode's derivation chain is self-contained. Stage I learns each task codebook via self-reconstruction on HQ images (Eq. 3), so the codebook prior is a trained representation, not an output defined by the final restoration. Stage II trains the latent diffusion denoiser to predict the Stage I HQ features z^n from LQ-conditioned noisy latents with Lstage2 = ||z^n - \hat z^n||_1 (Eq. 6); this is a standard supervised regression target, and the retrieved features are then decoded and used as an input to the Stage III restoration network. The final PSNR/SSIM/RMSE results in Tables 1 and 2 are computed on held-out test data and compared against eleven external baselines, so the central claim is falsifiable and not forced by construction. The only self-citation is [22] for the task-aware routing module; that module is an externally published architectural component and the paper's contribution does not reduce to it. The absence of direct retrieval-accuracy or distribution-alignment measurements weakens the mechanistic evidence for the diffusion component, but that is an evidence gap, not circularity.
Assumptions & free parameters
free parameters (7)
- codebook size per task (8192 entries, dimension 256) =
8192 x 256
- number of RQ iterations (8) =
8
- diffusion total time steps T (8) =
8
- beta schedule (linear from 0.1 to 0.99) =
beta_1=0.1, beta_T=0.99
- commitment loss weight delta (0.25) =
0.25
- number of experts in TARM (4) and top-k (1) =
4 experts, k=1
- NAF block configuration [2,2,4,4] and channels [64,128,256,256] =
NAF blocks, channels per level
assumptions (4)
- domain assumption The VQ codebook learned on clean images captures the full distribution of HQ medical image features necessary for restoration.
- domain assumption Latent diffusion with T=8 steps can reliably map a random Gaussian latent to the correct HQ prior feature when conditioned on LQ features.
- domain assumption The LQ condition encoder Elq provides sufficient information for the diffusion model to recover task-specific HQ features.
- domain assumption Using L1 losses only (reconstruction, diffusion, restoration) is sufficient to train a stable, high-quality restoration model.
invented entities (2)
-
Task-adaptive codebook bank
-
Latent diffusion-enhanced prior retrieval
Cite this review
Pith. "Pith review of All-in-One Medical Image Restoration with Latent Diffusion-Enhanced Vector-Quantized Codebook Prior." pith.science (2026). https://pith.science/paper/LOWQTOFN
@misc{pith2026250719874,
author = {Pith},
title = {Pith review of: All-in-One Medical Image Restoration with Latent Diffusion-Enhanced Vector-Quantized Codebook Prior},
year = {2026},
howpublished = {\url{https://pith.science/paper/LOWQTOFN}},
note = {Machine review of arXiv:2507.19874}
}
read the original abstract
All-in-one medical image restoration (MedIR) aims to address multiple MedIR tasks using a unified model, concurrently recovering various high-quality (HQ) medical images (e.g., MRI, CT, and PET) from low-quality (LQ) counterparts. However, all-in-one MedIR presents significant challenges due to the heterogeneity across different tasks. Each task involves distinct degradations, leading to diverse information losses in LQ images. Existing methods struggle to handle these diverse information losses associated with different tasks. To address these challenges, we propose a latent diffusion-enhanced vector-quantized codebook prior and develop \textbf{DiffCode}, a novel framework leveraging this prior for all-in-one MedIR. Specifically, to compensate for diverse information losses associated with different tasks, DiffCode constructs a task-adaptive codebook bank to integrate task-specific HQ prior features across tasks, capturing a comprehensive prior. Furthermore, to enhance prior retrieval from the codebook bank, DiffCode introduces a latent diffusion strategy that utilizes the diffusion model's powerful mapping capabilities to iteratively refine the latent feature distribution, estimating more accurate HQ prior features during restoration. With the help of the task-adaptive codebook bank and latent diffusion strategy, DiffCode achieves superior performance in both quantitative metrics and visual quality across three MedIR tasks: MRI super-resolution, CT denoising, and PET synthesis.
Figures
Reference graph
Works this paper leans on
-
[9]
IEEE transactions on medical imaging37(6), 1478–1487 (2018)
Kim, K., Wu, D., Gong, K., Dutta, J., Kim, J.H., Son, Y.D., Kim, H.K., El Fakhri, G., Li, Q.: Penalized pet reconstruction using deep learning prior and local linear fitting. IEEE transactions on medical imaging37(6), 1478–1487 (2018)
work page 2018
-
[1]
https://brain-development.org/ixi-dataset/
Ixi dataset. https://brain-development.org/ixi-dataset/
-
[2]
In: European conference on computer vision
Chen, L., Chu, X., Zhang, X., Sun, J.: Simple baselines for image restoration. In: European conference on computer vision. pp. 17–33. Springer (2022)
2022
-
[3]
In: International Conference on Medical Image Computing and Computer-Assisted Intervention
Chen, Z., Gao, Q., Zhang, Y., Shan, H.: Ascon: Anatomy-aware supervised con- trastive learning framework for low-dose ct denoising. In: International Conference on Medical Image Computing and Computer-Assisted Intervention. pp. 355–365. Springer (2023)
work page 2023
-
[4]
In: Proceedings of the IEEE/CVF winter conference on applications of computer vision
Georgescu, M.I., Ionescu, R.T., Miron, A.I., Savencu, O., Ristea, N.C., Verga, N., Khan, F.S.: Multimodal multi-head convolutional attention with various kernel sizes for medical image super-resolution. In: Proceedings of the IEEE/CVF winter conference on applications of computer vision. pp. 2195–2205 (2023)
2023
-
[5]
Advances in neural information processing systems33, 6840–6851 (2020)
Ho, J., Jain, A., Abbeel, P.: Denoising diffusion probabilistic models. Advances in neural information processing systems33, 6840–6851 (2020)
2020
-
[6]
Huang, Z., Zhang, J., Zhang, Y., Shan, H.: Du-gan: Generative adversarial net- workswithdual-domainu-net-baseddiscriminatorsforlow-dosectdenoising.IEEE Transactions on Instrumentation and Measurement71, 1–12 (2021)
2021
-
[7]
IEEE transactions on medical imaging 13(4), 601–609 (1994)
Hudson, H.M., Larkin, R.S.: Accelerated image reconstruction using ordered sub- sets of projection data. IEEE transactions on medical imaging 13(4), 601–609 (1994)
1994
Show all 31 references
-
[8]
IEEE transactions on medical imaging (2023) 10 Chen et al
Jang, S.I., Pan, T., Li, Y., Heidari, P., Chen, J., Li, Q., Gong, K.: Spach trans- former: Spatial and channel-wise transformer based on local and global self- attentions for pet image denoising. IEEE transactions on medical imaging (2023) 10 Chen et al
2023
-
[10]
In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition
Lee, D., Kim, C., Kim, S., Cho, M., Han, W.S.: Autoregressive image genera- tion using residual quantization. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 11523–11532 (2022)
2022
-
[11]
In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition
Li, B., Liu, X., Hu, P., Wu, Z., Lv, J., Peng, X.: All-in-one image restoration for unknown corruption. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 17452–17462 (2022)
2022
-
[12]
In: Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition
Li, G., Rao, C., Mo, J., Zhang, Z., Xing, W., Zhao, L.: Rethinking diffusion model for multi-contrast mri super-resolution. In: Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition. pp. 11365–11374 (2024)
2024
-
[13]
Medical Image Analysis77, 102335 (2022)
Luo, Y., Zhou, L., Zhan, B., Fei, Y., Zhou, J., Wang, Y., Shen, D.: Adaptive rectification based adversarial network with spectrum constraint for high-quality pet image synthesis. Medical Image Analysis77, 102335 (2022)
2022
-
[14]
In: Pro- ceedings of the IEEE/CVF conference on computer vision and pattern recognition
Luo, Z., Gustafsson, F.K., Zhao, Z., Sjölund, J., Schön, T.B.: Refusion: Enabling large-size realistic image restoration with latent-space diffusion models. In: Pro- ceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 1680–1691 (2023)
2023
-
[15]
https://doi.org/10.7937/9NPB-2637
McCollough, C., Chen, B., Holmes III, D.R., Duan, X., Yu, Z., Yu, L., Leng, S., Fletcher, J.: Low dose ct image and projection data (ldct-and-projection-data) (version 6) [data set] (2020). https://doi.org/10.7937/9NPB-2637
2020 doi
-
[16]
arXiv preprint arXiv:2409.13094 (2024)
Öztürk, Ş., Duran, O.C., Çukur, T.: Denomamba: A fused state-space model for low-dose ct denoising. arXiv preprint arXiv:2409.13094 (2024)
2024 arXiv
-
[17]
In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., Ommer, B.: High-resolution image synthesis with latent diffusion models. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 10684–10695 (2022)
2022
-
[18]
Outrageously large neural networks (2017)
Shazeer, N., Mirhoseini, A., Maziarz, K., Davis, A., Le, Q., Hinton, G., Dean, J.: The sparsely-gated mixture-of-experts layer. Outrageously large neural networks (2017)
2017
-
[19]
Advances in neural information processing systems30 (2017)
Van Den Oord, A., Vinyals, O., et al.: Neural discrete representation learning. Advances in neural information processing systems30 (2017)
2017
-
[20]
In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition
Wang, Z., Cun, X., Bao, J., Zhou, W., Liu, J., Li, H.: Uformer: A general u-shaped transformer for image restoration. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 17683–17693 (2022)
2022
-
[21]
In: Proceedings of the IEEE/CVF International Conference on Computer Vision
Xia, B., Zhang, Y., Wang, S., Wang, Y., Wu, X., Tian, Y., Yang, W., Van Gool, L.: Diffir: Efficient diffusion model for image restoration. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 13095–13105 (2023)
2023
-
[22]
In: International Con- ference on Medical Image Computing and Computer-Assisted Intervention
Yang, Z., Chen, H., Qian, Z., Yi, Y., Zhang, H., Zhao, D., Wei, B., Xu, Y.: All- in-one medical image restoration via task-adaptive routing. In: International Con- ference on Medical Image Computing and Computer-Assisted Intervention. pp. 67–77. Springer (2024)
2024
-
[23]
In: International Conference on Medical Image Computing and Computer-Assisted Intervention
Yang, Z., Zhou, Y., Zhang, H., Wei, B., Fan, Y., Xu, Y.: Drmc: A generalist model with dynamic routing for multi-center pet image synthesis. In: International Conference on Medical Image Computing and Computer-Assisted Intervention. pp. 36–46. Springer (2023)
2023
-
[24]
IEEE Transactions on Image Processing (2024) Title Suppressed Due to Excessive Length 11
Yao, M., Xu, R., Guan, Y., Huang, J., Xiong, Z.: Neural degradation representation learning for all-in-one image restoration. IEEE Transactions on Image Processing (2024) Title Suppressed Due to Excessive Length 11
2024
-
[25]
In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition
Zamir, S.W., Arora, A., Khan, S., Hayat, M., Khan, F.S., Yang, M.H.: Restormer: Efficient transformer for high-resolution image restoration. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 5728–5739 (2022)
2022
-
[26]
In: Proceedings of the IEEE/CVF con- ference on computer vision and pattern recognition
Zamir, S.W., Arora, A., Khan, S., Hayat, M., Khan, F.S., Yang, M.H., Shao, L.: Multi-stage progressive image restoration. In: Proceedings of the IEEE/CVF con- ference on computer vision and pattern recognition. pp. 14821–14831 (2021)
2021
-
[27]
In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition
Zhang, J., Huang, J., Yao, M., Yang, Z., Yu, H., Zhou, M., Zhao, F.: Ingredient- oriented multi-degradation learning for image restoration. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 5825– 5835 (2023)
2023
-
[28]
IEEE transactions on medical imaging40(3), 805–817 (2020)
Zhao, C., Dewey, B.E., Pham, D.L., Calabresi, P.A., Reich, D.S., Prince, J.L.: Smore: a self-supervised anti-aliasing and super-resolution algorithm for mri using deep learning. IEEE transactions on medical imaging40(3), 805–817 (2020)
2020
-
[29]
In: Proceedings of the European Conference on Computer Vision (ECCV)
Zhao, X., Li, H., Shen, X., Liang, X., Wu, Y.: A modulation module for multi- task learning with applications in image retrieval. In: Proceedings of the European Conference on Computer Vision (ECCV). pp. 401–416 (2018)
2018
-
[30]
IEEE transactions on image processing28(11), 5649–5662 (2019)
Zhao, X., Zhang, Y., Zhang, T., Zou, X.: Channel splitting network for single mr image super-resolution. IEEE transactions on image processing28(11), 5649–5662 (2019)
2019
-
[31]
Medical image analysis65, 101770 (2020)
Zhou, L., Schaefferkoetter, J.D., Tham, I.W., Huang, G., Yan, J.: Supervised learn- ing with cyclegan for low-dose fdg pet image denoising. Medical image analysis65, 101770 (2020)
2020
Reviewed August 15, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.