REVIEW 3 major objections 4 minor 20 references
Compressed Image Generation with Denoising Diffusion Codebook Models
T0 review · 3 major / 4 minor · reviewed 2026-08-09 · deepseek-v4-flash
Pith's one-line read The paper claims that diffusion sampling can be driven by tiny fixed codebooks of Gaussian vectors, preserving generation quality and giving every image a lossless bit-stream that also makes a state-of-the-art low-rate perceptual codec.
desk verdict A simple, clever mechanism—replacing Gaussian noise with fixed codebook vectors—that turns pretrained diffusion models into perceptual codecs; empirically strong, theoretically thin. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is a chain of fixed codebooks: for each reverse step $i$ there is a set $C_i$ of $K$ vectors sampled once from $\mathcal{N}(0,I)$ and never updated. Sampling uses $x_{i-1}=\mu_i(x_i)+\sigma_i C_i(k_i)$, so the only stochasticity is the index sequence; compression and conditional generation reduce to choosing $k_i$ by Eq. (7) or Eq. (8). The matching-pursuit variant refines each chosen noise as a convex combination of $M$ codebook entries to raise the bit rate, and Proposition 6.1 identifies the $K\to\infty$ limit of the posterior-matching loss with a probability-flow ODE, giving the selection rules a score-based interpretation.
What would settle it
A controlled ablation with the same fixed codebooks could, at a randomly chosen small fraction of steps, replace the greedy rule with the worst-matching codebook entry; if the decoded image's PSNR and FID barely move, the selection rule is not load-bearing, whereas a large drop would confirm that it is.
Extended reading notes
Core claim
DDCM replaces the standard Gaussian noise $z_i$ in the DDPM update with $C_i(k_i)$, a vector from a fixed codebook of $K$ i.i.d. Gaussian vectors per timestep, and uses the index sequence as the compressed representation. The paper shows empirically that this discrete process matches DDPM's FID at $K=64$ on ImageNet 256 and on MS-COCO with Stable Diffusion 2.1, and that as a codec it selects $k_i=\arg\max_{k}\langle C_i(k), x_0-\hat{x}_{0|i}\rangle$, achieving lower FID and LPIPS than prior methods at roughly $0.03$--$0.15$ bits per pixel. For conditional generation it chooses indices by minimizing a task loss; with the loss $\|C_i(k)-\sigma_i\nabla_{x_i}\log p_i(y|x_i)\|^2$, the paper proves that as $K\to\infty$ the process becomes a discretization of the probability-flow ODE over the posterior $p_0(x_0|y)$. Image compression is a special case with $y=x_0$, so the same codebook sampling produces both a sample and its bit-stream.
Load-bearing premise
The greedy per-step index rule is assumed to keep the whole trajectory close to the target, while the only proven statement is a $K\to\infty$ limit for a related loss; if locally optimal choices compound into large drift, the codec and restoration claims weaken.
Editorial extensions
If this is right
- Any pre-trained DDPM can be converted into a codec or conditional sampler without retraining, because only the sampling-time noise source and the index rule change.
- Every image produced by DDCM carries a lossless index bit-stream of $(T-1)\log_2(K)$ bits, so repeated decoding with the same indices reproduces it exactly.
- At low bit rates the codec scores lower FID and LPIPS than prior perceptual methods on Kodak, DIV2K, CLIC2020, and ImageNet while maintaining competitive PSNR.
- For inverse problems, restored images are generated together with compressed bit-streams, and the paper reports better perceptual quality than DPS and DDNM in colorization and $4\times$ super-resolution.
- With the posterior-matching loss, the $K\to\infty$ limit of the selection rule is a probability-flow ODE over the posterior, placing compression and conditional generation under one theoretical umbrella.
Reading between the lines
- The paper notes that DDCM indices form an image tokenization; a concrete next step it does not take is training a sequence model to predict those indices from text, which would make a pre-trained diffusion backbone text-conditional without fine-tuning it.
- Because the index rule is gradient-free and can optimize any per-image criterion, the same codec could be applied to non-differentiable quality metrics beyond NIQE and TOPIQ, and to video by coding residual index differences between frames.
- Entropy coding the index stream, learned dictionary codebooks, or shared codebooks across timesteps would cut bit rate without changing the core mechanism; the paper lists these as future work but does not quantify them.
- The $K\to\infty$ equivalence suggests a scaling study: measuring how FID and codec distortion vary as $K$ increases would quantify how redundant the continuous Gaussian noise space really is; the paper does not run that analysis.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces Denoising Diffusion Codebook Models (DDCM), which replace the continuous Gaussian noise draws in DDPM reverse sampling with entries from fixed, pre-sampled codebooks of iid Gaussian vectors (Eq. (6)). The authors show empirically that DDCM retains DDPM-level sample quality in both pixel and latent spaces even at codebook size K=64. They then turn DDCM into a lossy image codec by selecting, at each reverse step, the codebook entry with the largest inner product with the residual between the target image and the current denoised estimate (Eq. (7)). They report state-of-the-art perceptual compression at low bit rates across several datasets and compare favourably to many baselines. The framework is extended to compressed conditional generation, including image restoration and face restoration, with an accompanying asymptotic interpretation (Prop. 6.1) and several guidance variants. The paper includes code and a demo.
Significance. If the empirical claims hold, the paper makes a useful and surprising contribution: the continuous noise space of DDPMs is highly redundant, and a discrete codebook representation can simultaneously serve as a generative latent space and as a zero-shot lossy codec. The codec is practical, requires no training for the pre-trained diffusion model, and the index-stream representation is directly interpretable. The paper also provides a promising framework for compressed conditional generation. The main weaknesses are that the theoretical proposition is asymptotic and does not cover the finite-K greedy selection used in practice, and that the state-of-the-art claim is unqualified despite documented underperformance at high bit rates. The empirical results, if accompanied by the requested ablations and qualifications, would be a solid contribution.
major comments (3)
- [Sec. 6, Prop. 6.1; App. C.1.2] The proof of Proposition 6.1 derives the update xi−1 = xi − fi(xi) + g_i^2 ∇ log p_i(x_i|y) (Eq. (28)) and calls it a discretization of the probability flow ODE (Eq. (18)). However, the probability flow ODE in Eq. (18) has score coefficient g^2/2, not g^2. The derived update is a deterministic discretization of the reverse-time SDE (Eq. (17)) with the stochastic term replaced by its conditional mean, not a discretization of the probability flow ODE. As written, the proposition is not established; please correct the statement or adjust the coefficient, or rephrase it as an approximation of the reverse-time SDE with zeroed noise.
- [Sec. 5, Eq. (7); Sec. 7] The compression codec's success rests on the greedy per-step selection rule (7), but the paper provides no finite-K analysis of this rule. Proposition 6.1 covers the related loss (9) only in the K→∞ limit, and Sec. 7 explicitly disclaims a theoretical explanation for the selection strategies. In the operating regime (e.g., K=16, latent dimension n≈10^4), the expected cosine similarity between a selected codebook vector and the residual is about sqrt(2 log K / n) ≈ 0.02, so the per-step signal is weak, and errors can compound over T=1000 steps. I recommend adding an ablation that compares Eq. (7) with a one-step look-ahead or short-horizon selection on at least one dataset; this would indicate whether the reported rate-distortion-perception numbers depend on the unproven compounding assumption.
- [Abstract; Sec. 5] The abstract's claim of 'state-of-the-art perceptual image compression results' is unqualified, but the paper's own results in Fig. 5 and Tables 2–5 show that at the highest bit rates DDCM underperforms several baselines (for example, on Kodak24, PerCo (SD) and CRDR-R achieve lower FID and higher PSNR than Ours at approximately 0.13–0.15 BPP in Table 3). Please qualify the claim to the low-to-mid bit-rate regime in the abstract and in the conclusions.
minor comments (4)
- [App. B.5, Eqs. (13)–(15)] The text describes the matching-pursuit refinement as a 'convex combination' of codebook entries, but after the renormalization in Eq. (15) the transmitted coefficients no longer form a convex combination; please rephrase or justify the terminology.
- [Sec. 5, bit-rate formula] The bit-stream length formula (T−1)(log2(K)M + C(M−1)) treats C as a number of bits per coefficient, while App. B.5 defines C as the number of quantization levels; the formula should involve ceil(log2 C), or the text should clearly state that C is the number of bits per coefficient.
- [Tables 2–7] The paper reports all metrics as single point estimates without stating whether they come from one evaluation or are averaged over multiple runs or seeds; for FID computed over random subsets, please provide the number of subsets or confidence intervals, or explicitly note that the codebooks are fixed and the evaluations are deterministic.
- [App. C.2, Eq. (37)] In the derivation of the equivalence between Eq. (7) and the L_P loss, the factor σ_i√(ᾱ_i)/(1−ᾱ_i) appears in Eq. (37) but is dropped inconsistently before reaching Eq. (44); please verify the algebra or define the scaling convention explicitly.
Circularity Check
No circular reduction found; the codec and sampling claims are evaluated against external baselines, and the theoretical Proposition 6.1 is an explicit consistency interpretation rather than a derivation of finite-K success.
full rationale
The derivation chain is self-contained. The index selection rule in Eq. (7) is an explicit, greedy rule defined by the paper and then evaluated empirically against external codecs; it is not a fitted parameter renamed as a prediction. Proposition 6.1 is a limit statement: when L is set to LP in Eq. (9), the chosen codebook vector converges to the posterior score correction as K tends to infinity, so the process becomes a posterior probability-flow ODE. This follows directly from the definition of LP and is honestly labeled a mathematical interpretation; the paper explicitly disclaims any theoretical explanation of finite-K behavior in Sec. 7 ('our work does not explain theoretically why DDCM sampling and our simple index selection strategies work so effectively'). That disclaimer forecloses any reading of Prop. 6.1 as the source of the empirical compression results. The only self-citations are comparison baselines from the same group, notably PSC (Elata et al., 2024) and PMRF (Ohayon et al., 2025), but the central state-of-the-art claims rest on external baselines such as BPG, HiFiC, PerCo (SD), ILLM, CRDR, and IPIC. These self-citations are not load-bearing, so the paper contains no significant circularity; at most, there is minor non-load-bearing self-citation.
Assumptions & free parameters
free parameters (5)
- Codebook size K =
varies: 2,4,8,16,64 (generation); 16 to 8192 (compression)
- Number of sampling timesteps T =
300, 500, or 1000 depending on experiment
- Matching pursuit parameters M and C =
M in {2,3,4,5,6}; C=3
- Perception-distortion weight lambda =
1
- Guidance scales =
CFG 3 for latent evaluation; classifier guidance 1 for pixel evaluation
assumptions (6)
- standard math A well-trained score model s_i(x_i) = ∇ log p_i(x_i) makes the DDPM reverse update (Eq. 2) a valid sampler from p0.
- standard math K iid N(0,I) codebook vectors become dense in R^n as K→∞, so the minimum distance to any fixed vector tends to 0.
- standard math High-dimensional codebook vectors have nearly constant squared norm (chi-squared concentration), so Eq. (7) approximates the LP loss.
- domain assumption The conditional likelihood p_i(y|x_i) is approximated as a multivariate Gaussian centered at A x_i, making Eq. (10) a proxy for the posterior score.
- domain assumption The face restoration method assumes r(y) ≈ E[x0|y] is a good MMSE approximation, allowing test-time optimization without ground truth.
- domain assumption Pre-trained DDMs and VAEs are used as fixed components; their approximation errors bound the codec's distortion.
invented entities (1)
-
Fixed noise codebooks C_i
Cite this review
Pith. "Pith review of Compressed Image Generation with Denoising Diffusion Codebook Models." pith.science (2026). https://pith.science/paper/EII7SGHC
@misc{pith2026250201189,
author = {Pith},
title = {Pith review of: Compressed Image Generation with Denoising Diffusion Codebook Models},
year = {2026},
howpublished = {\url{https://pith.science/paper/EII7SGHC}},
note = {Machine review of arXiv:2502.01189}
}
read the original abstract
We present a novel generative approach based on Denoising Diffusion Models (DDMs), which produces high-quality image samples along with their losslessly compressed bit-stream representations. This is obtained by replacing the standard Gaussian noise sampling in the reverse diffusion with a selection of noise samples from pre-defined codebooks of fixed iid Gaussian vectors. Surprisingly, we find that our method, termed Denoising Diffusion Codebook Model (DDCM), retains sample quality and diversity of standard DDMs, even for extremely small codebooks. We leverage DDCM and pick the noises from the codebooks that best match a given image, converting our generative model into a highly effective lossy image codec achieving state-of-the-art perceptual image compression results. More generally, by setting other noise selections rules, we extend our compression method to any conditional image generation task (e.g., image restoration), where the generated images are produced jointly with their condensed bit-stream representations. Our work is accompanied by a mathematical interpretation of the proposed compressed conditional generation schemes, establishing a connection with score-based approximations of posterior samplers for the tasks considered.
Figures
Figures from the paper (24 more)
Reference graph
Works this paper leans on
-
[6]
2 Kang, B., Tripathi, S., and Nguyen, T
doi: 10.1145/3581783.3611694. 2 Kang, B., Tripathi, S., and Nguyen, T. Q. Toward joint im- age generation and compression using generative adver- sarial networks. arXiv, 2019. URL https://arxiv. org/abs/1901.07838. 2 Karras, T., Aila, T., Laine, S., and Lehtinen, J. Progres- sive growing of GANs for improved quality, stability, and variation. In Internati...
-
[8]
2, 45 Meng, C., He, Y ., Song, Y ., Song, J., Wu, J., Zhu, J.- Y ., and Ermon, S
URL https://proceedings.mlr.press/ v235/manor24a.html. 2, 45 Meng, C., He, Y ., Song, Y ., Song, J., Wu, J., Zhu, J.- Y ., and Ermon, S. SDEdit: Guided image synthesis and editing with stochastic differential equations. In International Conference on Learning Representations,
-
[9]
URL https://openreview.net/forum? id=aBsCjcPu_tE. 1, 2, 45 Mentzer, F., Toderici, G. D., Tschannen, M., and Agustsson, E. High-fidelity generative image compression. Advances in Neural Information Processing Systems , 33:11913– 11924, 2020. 2, 5 Mittal, A., Soundararajan, R., and Bovik, A. C. Making a “completely blind” image quality analyzer. IEEE Signal...
-
[10]
7 Ohayon, G., Michaeli, T., and Elad, M
URL https://proceedings.mlr.press/ v202/ohayon23a.html. 7 Ohayon, G., Michaeli, T., and Elad, M. Posterior-mean recti- fied flow: Towards minimum MSE photo-realistic image restoration. In The Thirteenth International Conference on Learning Representations, 2025. URL https:// openreview.net/forum?id=hPOt3yUXii. 7, 8 Oquab, M., Darcet, T., Moutakanni, T., V...
work page 2025
-
[11]
43 Rajesh, B., Dusa, N., Javed, M., Dubey, S
URL https://proceedings.mlr.press/ v139/radford21a.html. 43 Rajesh, B., Dusa, N., Javed, M., Dubey, S. R., and Nagab- hushan, P. T2ci-gan: Text to compressed image genera- tion using generative adversarial network. In Gupta, D., Bhurchandi, K., Murala, S., Raman, B., and Kumar, S. (eds.), Computer Vision and Image Processing, pp. 292– 307, Cham, 2023. Spr...
arXiv 2023
-
[14]
2 Theis, L., Shi, W., Cunningham, A., and Husz´ar, F
URL https://proceedings.mlr.press/ v162/theis22a.html. 2 Theis, L., Shi, W., Cunningham, A., and Husz´ar, F. Lossy image compression with compressive autoencoders. In International Conference on Learning Representations,
-
[16]
RDPM: Solve Diffusion Probabilistic Models via Recurrent Token Prediction
URL https://openreview.net/forum? id=SyxAb30cY7. 40 Tumanyan, N., Geyer, M., Bagon, S., and Dekel, T. Plug- and-play diffusion features for text-driven image-to- image translation. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pp. 1921–1930, 2023. 45 Vincent, P. A connection between score matching and de- noising ...
work page Pith review arXiv 1921
- [17]
Show all 20 references
-
[19]
on the same grounds
(23) 32 Compressed Image Generation with Denoising Diffusion Codebook Models Thus, Ci(ki) − → K→∞ gi∇xi log pi(y|xi). (24) Plugging Eq. (24) into Eq. (20), we get xi−1 − → K→∞ xi − fi(xi) + g2 i si(xi) + g2 i ∇xi log pi(y|xi) (25) =xi − fi(xi) + g2 i ∇xi log pi(xi) + g2 i ∇xi ...
2020
-
[20]
Figure 26 shows that our CCFG method is on par with CFG in terms of FID, while CFG produces higher CLIP scores
with the OpenAI CLIP ViT-L/14 model (Radford et al., 2021). Figure 26 shows that our CCFG method is on par with CFG in terms of FID, while CFG produces higher CLIP scores. This suggests that the outputs of CFG better align with the input text prompts compared to CCFG. Yet, the...
2024
-
[1981]
C., Hosseinzadeh, R., Sui, Y ., Ross, B
3 Stein, G., Cresswell, J. C., Hosseinzadeh, R., Sui, Y ., Ross, B. L., Villecroze, V ., Liu, Z., Caterini, A. L., Taylor, E., and Loaiza-Ganem, G. Exposing flaws of generative model evaluation metrics and their unfair treatment of dif- fusion models. In Thirty-seventh Confere...
2023 arXiv
-
[2014]
cc/paper_files/paper/2014/file/ 5ca3e9b122f61f8f06494c97b1afccf3-Paper
URL https://proceedings.neurips. cc/paper_files/paper/2014/file/ 5ca3e9b122f61f8f06494c97b1afccf3-Paper. pdf. 2 Gu, Y ., Wang, X., Xie, L., Dong, C., Li, G., Shan, Y ., and Cheng, M.-M. Vqfr: Blind face restoration with vector-quantized dictionary and parallel decoder. In Comp...
2014
-
[2017]
2 Theis, L., Salimans, T., Hoffman, M
URL https://openreview.net/forum? id=rJiNwv9gg. 2 Theis, L., Salimans, T., Hoffman, M. D., and Mentzer, F. Lossy compression with gaussian diffusion. arXiv preprint arXiv:2206.08889, 2022. 1, 2 13 Compressed Image Generation with Denoising Diffusion Codebook Models Toderici, G...
2022 arXiv
-
[2019]
cc/paper_files/paper/2019/file/ 3001ef257407d5a371a96dcd947c7d93-Paper
URL https://proceedings.neurips. cc/paper_files/paper/2019/file/ 3001ef257407d5a371a96dcd947c7d93-Paper. pdf. 3 Song, Y ., Sohl-Dickstein, J., Kingma, D. P., Kumar, A., Er- mon, S., and Poole, B. Score-based generative modeling through stochastic differential equations. In Int...
2019
-
[2021]
cc/paper_files/paper/2021/file/ 7d6044e95a16761171b130dcb476a43e-Paper
URL https://proceedings.neurips. cc/paper_files/paper/2021/file/ 7d6044e95a16761171b130dcb476a43e-Paper. pdf. 7 Jiang, W. and Wang, R. MLIC++: Linear complex- ity multi-reference entropy modeling for learned image compression. In ICML 2023 Workshop Neural Com- pression: From I...
2021
-
[2022]
ISBN 978-3-031-19796-3
Springer-Verlag. ISBN 978-3-031-19796-3. doi: 10 Compressed Image Generation with Denoising Diffusion Codebook Models 10.1007/978-3-031-19797-0 8. URL https://doi. org/10.1007/978-3-031-19797-0_8 . 7 Hertz, A., Aberman, K., and Cohen-Or, D. Delta denoising score. In Proceeding...
-
[2023]
2 Jiang, W., Yang, J., Zhai, Y ., Ning, P., Gao, F., and Wang, R
URL https://openreview.net/forum? id=hxIpcSoz2t. 2 Jiang, W., Yang, J., Zhai, Y ., Ning, P., Gao, F., and Wang, R. MLIC: Multi-reference entropy model for learned image compression. In Proceedings of the 31st ACM International Conference on Multimedia, pp. 7618–7627,
-
[2024]
2, 7 Liu, H., Zhang, G., Chen, J., and Khisti, A. J. Lossy com- pression with distribution shift as entropy constrained optimal transport. In International Conference on Learn- ing Representations, 2021. 2 Lugmayr, A., Danelljan, M., Romero, A., Yu, F., Timofte, R., and Van Go...
2021 arXiv
-
[2025]
2 Yang, R
URL https://openreview.net/forum? id=xNsIfzlefG. 2 Yang, R. and Mandt, S. Lossy image compression with conditional diffusion models. In Oh, A., Naumann, T., Globerson, A., Saenko, K., Hardt, M., and Levine, S. (eds.), Advances in Neural Information Processing Sys- tems, volume...
2020
-
[6851]
1, 3, 32, 33 Huang, G
Curran Associates, Inc., 2020. 1, 3, 32, 33 Huang, G. B., Mattar, M., Berg, T., and Learned-Miller, E. Labeled Faces in the Wild: A Database forStudy- ing Face Recognition in Unconstrained Environments. In Workshop on Faces in ’Real-Life’ Images: Detec- tion, Alignment, and Re...
2020
Reviewed August 9, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.