REVIEW 3 major objections 3 minor 39 references
Making Images from Images: Interleaving Denoising and Transformation
T0 review · 3 major / 3 minor · reviewed 2026-08-12 · deepseek-v4-flash
Pith's one-line read The paper claims that interleaving diffusion denoising with a dynamic tile-matching step lets any fixed source image, such as the Mona Lisa, be rearranged into a new image matching a text prompt.
desk verdict A genuinely new dynamic-matching twist on Visual Anagrams, with strong qualitative results, but the latent-space 2% rollout mixing leaves the central interleaving claim under-supported. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the interleaved dynamic matching loop: at each diffusion step, a rollout lookahead produces an idealized image for each prompt, and a Kuhn-Munkres assignment solves $\min_P \operatorname{tr}(PD)$ to pair tiles by L2 pixel distance, yielding the next permutation $\psi_{t-1}$. This turns a previously static, hand-chosen transform into a learned, iteration-dependent one; for a fixed source image $\beta$, rollout mixing $w^{\text{image}}_t \Psi^i_t(\beta) + w^{\text{noise}}_t \epsilon$ converts the noise-free source into a valid diffusion input.
What would settle it
Run the fixed-source pipeline on the same prompts and seeds while replacing the rollout with the current noisy image or with random permutations; if the resulting images keep matching prompts as well on CLIP scores, the rollout is not doing the claimed work. Alternatively, sweep the rollout length from 1 to 50 and check whether quality is flat; a flat curve would contradict the claim that lookahead is the mechanism.
Extended reading notes
Core claim
The central claim is that the transform between images need not be fixed in advance: a permutation of image tiles can be learned jointly with the image content, enabling a static source image to become a prompt-specified subject. The paper formulates this as a constrained optimization and solves it by alternating diffusion steps with dynamic matching: a short rollout diffusion run predicts an idealized denoised image, the idealized image is matched to the source tiles by the Hungarian algorithm, and the resulting permutation is used in the next diffusion step. For a fixed source, the mainline image is simply the permuted source, so the only free variables are the permutations themselves. In latent diffusion, rollouts run the full 50-step process from a 2% mixture of image and noise, and matching is done after decoding to pixels. The authors report that larger tile counts improve results and that the same interleaving works for concentric-ring rotations and flips, not only tile permutations.
Load-bearing premise
The whole procedure stands on the assumption that a short rollout from the current image reliably predicts what the final denoised image will look like; the lookahead length of 5 steps and the latent mixing ratio of 2% are empirical choices with no guarantee of convergence.
Editorial extensions
If this is right
- Any fixed image can serve as the palette: the Mona Lisa, Starry Night, or a user's own photo can be turned into a new subject using only its exact tiles.
- Finer tile grids give more degrees of freedom, so quality improves as the number of tiles grows from 8x8 to 64x64, opposite to static-transform baselines.
- The same alternating denoising and assignment procedure extends beyond tile permutations to independent rotations of concentric rings and to flips and rotations at multiple resolutions.
- With c copies of the source, possibly infinite, tiles may be reused, and multiple different source images can be mixed, enabling finer control over which source tiles appear.
- In latent diffusion, dynamic matching plus full-length rollouts yields lower CLIP distance to prompts than pixel-space DeepFloyd results, meaning the method carries over to modern systems.
Reading between the lines
- The rollout lookahead is effectively a planning step: it guesses where each prompt would go before committing a permutation. The same pattern could be applied to other discrete generative constraints, such as collage layout or puzzle assembly, wherever a cheap forward model exists.
- Because more tiles improve quality, the method's limit may be set by the matching optimizer rather than image content; very large tile counts would test whether Hungarian matching remains tractable and whether perceptual distances beat L2.
- The constraint-propagation effect visible with multiple prompts suggests the method could be used to co-generate a set of images that share hidden structure, potentially for visual cryptography or adversarial illusions for vision models.
- A direct ablation replacing rollouts with the current noisy image would isolate whether the lookahead or the matching itself is responsible for the gains; the paper's design suggests lookahead is essential but does not prove it.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes an iterative method for generating images by rearranging tiles of a fixed source image. It builds on Visual Anagrams and interleaves diffusion denoising with dynamic matching (Hungarian algorithm) on rollout images to update tile permutations. The method is demonstrated in both pixel space (DeepFloyd) and latent space (Stable Diffusion 2.1), with extensions to multiple copies of the source, multiple source images, and alternative transformations such as concentric rings and flips. The main claims are that dynamic matching improves over static matching, that a pre-specified source image can be transformed into a novel subject, and that increasing the number of tiles makes the problem easier.
Significance. If the central claims hold, the paper is a useful contribution to controlled diffusion-based image generation and optical-illusion synthesis. The general idea of interleaving diffusion steps with an energy-minimization matching step is interesting and goes beyond previous tile-permutation work by removing the need for pre-specified transformations. The paper includes a large quantitative study (1200 CLIP-distance trials with Wilcoxon tests) and a broad set of qualitative demonstrations, which are strengths. However, the quantitative evidence is thin for the headline fixed-source latent-diffusion claim, and the iterative feedback mechanism is not directly validated in that setting.
major comments (3)
- [Section 4, Eq. (2), Figure 10] The central fixed-source latent claim depends on the 2% rollout mixing ratio, yet the paper provides no measurement that the current source permutation Ψ_t(β) influences the idealized images Q_i^t. At 2% image weight, the rollout input is dominated by fresh noise and the prompt, so Q_i^t may be effectively independent of Ψ_t(β); in that case the iterative dynamic matching is a one-shot approximation of β by a prompt-conditioned diffusion sample, and the interleaving loop is not doing the claimed work. Please add a direct ablation: vary the mixing ratio, compare final outputs after one iteration versus many iterations, and/or measure how much source information survives (for example, by correlating final tiles with the source permutation). This is load-bearing because the headline result is that a fixed source image can be rearranged into a novel subject.
- [Section 2.1] The quantitative evaluation reports statistically significant but extremely small CLIP-distance improvements (22.39 vs 22.43 for 4×4 tiles and 22.36 vs 22.40 for 8×8 tiles) without effect sizes, variances, or a discussion of practical significance. More importantly, this evaluation is for the two-prompt dynamic-matching case, not for the fixed-source latent-diffusion setting that is the paper's main claim. The fixed-source results in Section 4.1 are supported only by qualitative figures, with no comparison against a one-shot baseline (for example, generate a prompt-conditioned image and directly match β's tiles to it). Please add quantitative evaluation for the fixed-source latent case, including such a baseline.
- [Section 3, penultimate paragraph] The assertion that with a fixed β, N prompts need not be solved simultaneously and that independent solves yield 'equivalent results' is made without proof or demonstration. This is logically important: if true, the parallel constraint propagation shown in Figure 13 is not needed for fixed-source tasks; if false, the multi-prompt fixed-source results are not supported. Please provide a derivation or a controlled comparison between simultaneous and independent solves.
minor comments (3)
- [Section 4.1] The final sentence says the final outputs are the images Ψ_0(x_0), but Section 3 eliminated x_t for the fixed-source formulation; this should be clarified or corrected.
- [Figure 12] The comparison uses different prompt prefixes ('a painting' for Visual Anagrams, 'a photograph of' for the proposed method), which confounds the qualitative comparison.
- [Throughout] There are typos such as 'the easier it is our for system' (Section 1) and 'Analagous' (Section 4.1), and the abstract's claim that any existing image can be transformed is stronger than Section 3's acknowledged limitation for low-color sources.
Circularity Check
No circularity: the interleaving algorithm iteratively optimizes a permutation against diffusion rollouts; no prediction reduces by construction to a fitted parameter.
full rationale
The paper's derivation chain is an iterative optimization, not a closed-form derivation. The permutation ψ_t is updated by dynamic matching between tiles of the source β and the rollout output Q_t, where Q_t is produced by a diffusion process seeded with the current Ψ_t(β) and conditioned on the external prompt. The source image and prompt are inputs; the CLIP distance used for quantitative evaluation is an external metric against the prompts, not the fitted matching cost. No equation defines the predicted output as the fitted value by construction: the lookahead length l=5 and the 2% rollout mixing ratio are empirical hyperparameters, not fitted parameters renamed as predictions. There are no self-citations or author-imported uniqueness theorems; the only cited prior framework (Visual Anagrams, [12]) is external and used as a baseline. The 'chicken-and-egg' passage in Section 2.1 is an explicit statement of the joint optimization that the alternating algorithm addresses; it is not a circular definition. Concerns that the 2% mixing ratio may make latent rollouts nearly independent of the current source permutation, or that convergence is only empirically observed, are correctness and robustness issues rather than circularity.
Assumptions & free parameters
free parameters (3)
- rollout lookahead steps l =
5
- rollout mixing ratio =
2%
- mainline steps T =
15
assumptions (3)
- domain assumption Diffusion models (DDPM) with classifier-free guidance produce images matching text prompts
- domain assumption L2 distance between image tiles is an adequate measure of tile similarity
- ad hoc to paper The rollout lookahead approximates the final denoised image well enough for matching
Cite this review
Pith. "Pith review of Making Images from Images: Interleaving Denoising and Transformation." pith.science (2026). https://pith.science/paper/JPOKLIZ2
@misc{pith2026241115925,
author = {Pith},
title = {Pith review of: Making Images from Images: Interleaving Denoising and Transformation},
year = {2026},
howpublished = {\url{https://pith.science/paper/JPOKLIZ2}},
note = {Machine review of arXiv:2411.15925}
}
read the original abstract
Simply by rearranging the regions of an image, we can create a new image of any subject matter. The definition of regions is user definable, ranging from regularly and irregularly-shaped blocks, concentric rings, or even individual pixels. Our method extends and improves recent work in the generation of optical illusions by simultaneously learning not only the content of the images, but also the parameterized transformations required to transform the desired images into each other. By learning the image transforms, we allow any source image to be pre-specified; any existing image (e.g. the Mona Lisa) can be transformed to a novel subject. We formulate this process as a constrained optimization problem and address it through interleaving the steps of image diffusion with an energy minimization step. Unlike previous methods, increasing the number of regions actually makes the problem easier and improves results. We demonstrate our approach in both pixel and latent spaces. Creative extensions, such as using infinite copies of the source image and employing multiple source images, are also given.
Figures
Figures from the paper (14 more)
Reference graph
Works this paper leans on
-
[1]
In: Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition
Amir, D., Weiss, Y.: Understanding and simplifying perceptual distances. In: Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 12226–12235 (2021) 5
work page 2021
-
[2]
In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition
Avrahami, O., Lischinski, D., Fried, O.: Blended diffusion for text-driven editing of natural images. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 18208–18218 (2022) 2
work page 2022
-
[3]
Bar-Tal, O., Yariv, L., Lipman, Y., Dekel, T.: Multidiffusion: Fusing diffusion paths for controlled image generation (2023) 3
work page 2023
-
[4]
Burgert, R., Li, X., Leite, A., Ranasinghe, K., Ryoo, M.S.: Diffusion illusions: Hiding images in plain sight (2023) 2
work page 2023
-
[5]
Chen, Z., Geng, D., Owens, A.: Images that sound: Composing images and sounds on a single canvas (2024) 2
work page 2024
-
[6]
IEEE Transactions on Pattern Analysis and Machine Intelligence (2023) 2
Croitoru, F.A., Hondru, V., Ionescu, R.T., Shah, M.: Diffusion models in vision: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence (2023) 2
work page 2023
-
[7]
J Psychol Cognition3(2) (2018) 1
Das, K.K.: Role of pre-embedded associated memory in generation of optical illusions. J Psychol Cognition3(2) (2018) 1
work page 2018
-
[8]
Advances in neural information processing systems34, 8780–8794 (2021) 2
Dhariwal, P., Nichol, A.: Diffusion models beat gans on image synthesis. Advances in neural information processing systems34, 8780–8794 (2021) 2
work page 2021
Show all 39 references
-
[9]
In: Bengio, S., Wallach, H., Larochelle, H., Grauman, K., Cesa- Bianchi, N., Garnett, R
Elsayed, G., Shankar, S., Cheung, B., Papernot, N., Kurakin, A., Goodfellow, I., Sohl-Dickstein, J.: Adversarial examples that fool both computer vision and time-limited humans. In: Bengio, S., Wallach, H., Larochelle, H., Grauman, K., Cesa- Bianchi, N., Garnett, R. (eds.) Adv...
2018
-
[10]
In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)
Everaert, M.N., Bocchio, M., Arpa, S., Süsstrunk, S., Achanta, R.: Diffusion in style. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). pp. 2251–2261 (October 2023) 2
2023
-
[11]
ACM Siggraph Computer Graphics25(4), 27–30 (1991) 2
Freeman, W.T., Adelson, E.H., Heeger, D.J.: Motion without movement. ACM Siggraph Computer Graphics25(4), 27–30 (1991) 2
1991
-
[12]
arXiv:2311.17919 (November 2023),https://arxiv
Geng, D., Park, I., Owens, A.: Visual anagrams: Generating multi-view optical illusions with diffusion models. arXiv:2311.17919 (November 2023),https://arxiv. org/abs/2311.17919 2, 4, 5, 6, 11, 14
2023 arXiv
-
[13]
Geng, D., Park, I., Owens, A.: Factorized diffusion: Perceptual illusions by noise decomposition (2024) 2, 3
2024
-
[14]
arXiv preprint arXiv:1412.6572 (2014) 1
Goodfellow, I.J., Shlens, J., Szegedy, C.: Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6572 (2014) 1
2014 arXiv
-
[15]
In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
Hendrycks, D., Zhao, K., Basart, S., Steinhardt, J., Song, D.: Natural adversarial examples. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 15262–15271 (June 2021) 1
2021
-
[16]
Ho, J., Jain, A., Abbeel, P.: Denoising diffusion probabilistic models. In: Ad- vances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual (2020), https : / / proceedings . neu...
2020
-
[17]
arXiv:2207.12598 (2022) 4
Ho, J., Salimans, T.: Classifier-free diffusion guidance. arXiv:2207.12598 (2022) 4
2022 arXiv
-
[18]
ACM Trans
Hsiao, K.W., Huang, J.B., Chu, H.K.: Multi-view wire art. ACM Trans. Graph. 37(6), 242 (2018) 2 Making Images from Images: Interleaving Denoising and Transformation 23
2018
-
[19]
Jaini, P., Clark, K., Geirhos, R.: Intriguing properties of generative classifiers (2024) 1
2024
-
[20]
Naval research logistics quarterly2(1-2), 83–97 (1955) 4, 6
Kuhn, H.W.: The hungarian method for the assignment problem. Naval research logistics quarterly2(1-2), 83–97 (1955) 4, 6
1955
-
[21]
Liu, N., Li, S., Du, Y., Torralba, A., Tenenbaum, J.B.: Compositional visual generation with composable diffusion models (2023) 2
2023
-
[22]
UCLA USJ24 (2011) 1
Lo, C., Dinov, I.: Investigation of optical illusions on the aspects of gender and age. UCLA USJ24 (2011) 1
2011
-
[23]
https://midjourney.com (2024), accessed: 2024-5-22 2
Midjourney: Midjourney. https://midjourney.com (2024), accessed: 2024-5-22 2
2024
-
[24]
ACM Transactions on Graphics28(5), 156–1 (2009) 2
Mitra, N.J., Pauly, M.: Shadow art. ACM Transactions on Graphics28(5), 156–1 (2009) 2
2009
-
[25]
Journal of the society for industrial and applied mathematics5(1), 32–38 (1957) 4, 6
Munkres, J.: Algorithms for the assignment and transportation problems. Journal of the society for industrial and applied mathematics5(1), 32–38 (1957) 4, 6
1957
-
[26]
Ngo, J., Sankaranarayanan, S., Isola, P.: Is CLIP fooled by optical illusions? (2023), https://openreview.net/forum?id=YdGkE4Ugg2C 1
2023
-
[27]
arXiv preprint arXiv:2112.10741 (2021) 2
Nichol,A.,Dhariwal,P.,Ramesh,A.,Shyam,P.,Mishkin,P.,McGrew,B.,Sutskever, I., Chen, M.: Glide: Towards photorealistic image generation and editing with text- guided diffusion models. arXiv preprint arXiv:2112.10741 (2021) 2
2021 arXiv
-
[28]
Oliva,A.,Torralba,A.,Schyns,P.G.:Hybridimages.ACMTransactionsonGraphics (TOG) 25(3), 527–532 (2006) 1, 2
2006
-
[29]
In: International Conference on Machine Learning
Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al.: Learning transferable visual models from natural language supervision. In: International Conference on Machine Learning. pp. 8748–8763. PMLR (2021) 6, 8
2021
-
[30]
In: International Conference on Machine Learning
Ramesh, A., Pavlov, M., Goh, G., Gray, S., Voss, C., Radford, A., Chen, M., Sutskever, I.: Zero-shot text-to-image generation. In: International Conference on Machine Learning. pp. 8821–8831. PMLR (2021) 2
2021
-
[31]
In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., Ommer, B.: High-resolution image synthesis with latent diffusion models. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 10684–10695 (2022) 11
2022
-
[32]
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., Ommer, B.: High-resolution image synthesis with latent diffusion models (2021) 2, 9
2021
-
[33]
https://doi.org/10.48550/ARXIV.2205.11487 , https: //arxiv.org/abs/2205.11487 2
Saharia, C., Chan, W., Saxena, S., Li, L., Whang, J., Denton, E., Ghasemipour, S.K.S., Ayan, B.K., Mahdavi, S.S., Lopes, R.G., Salimans, T., Ho, J., Fleet, D.J., Norouzi, M.: Photorealistic text-to-image diffusion models with deep language understanding (2022). https://doi.org...
-
[34]
Advances in neural information processing systems35, 36479–36494 (2022) 2
Saharia, C., Chan, W., Saxena, S., Li, L., Whang, J., Denton, E.L., Ghasemipour, K., Gontijo Lopes, R., Karagol Ayan, B., Salimans, T., et al.: Photorealistic text- to-image diffusion models with deep language understanding. Advances in neural information processing systems35,...
2022
-
[35]
Shonenkov, A., Konstantinov, M., Bakshandaeva, D., Schuhmann, C., Ivanova, K., Klokova, N.: If by deepfloyd lab at stabilityai.https://github.com/deep-floyd/IF (2023), accessed: 2024-5-22 2, 4
2023
-
[36]
arXiv preprint arXiv:2010.02502 (2020) 2
Song, J., Meng, C., Ermon, S.: Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502 (2020) 2
2020 arXiv
-
[37]
Tancik, M.: Illusion-diffusion.https://github.com/tancik/Illusion-Diffusion (2023), accessed: 2024-5-22 2
2023
-
[38]
In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition
Tumanyan, N., Geyer, M., Bagon, S., Dekel, T.: Plug-and-play diffusion features for text-driven image-to-image translation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 1921–1930 (2023) 2 24 S. Baluja et al
2023
-
[39]
New Ideas in Psychology36, 1–9 (2015) 1
de Wit, M.M., van der Kamp, J., Withagen, R.: Visual illusions and direct percep- tion: Elaborating on gibson’s insights. New Ideas in Psychology36, 1–9 (2015) 1
2015
Reviewed August 12, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.