REVIEW 4 major objections 6 minor 18 references
Align & Invert: Solving Inverse Problems with Diffusion and Flow-based Models via Representation Alignment
T0 review · 4 major / 6 minor · reviewed 2026-08-03 · deepseek-v4-flash
Pith's one-line read Adding representation alignment to diffusion-based inverse solvers improves perceptual quality and cuts inference steps.
desk verdict A genuine new application of REPA as an inference-time regularizer for latent inverse solvers; the core empirical claim holds for Latent DPS, but the ReSample comparison is confounded and the evaluation needs tightening. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the REPA regularizer: a patch-wise cosine alignment between the diffusion model's intermediate hidden states ht = DIFFENC(xt, t) and DINOv2 features of a proxy reconstruction, c_proxy = f_DINOv2(bar x0). It is injected as an extra gradient in the reverse process, alongside the measurement-matching gradient. Its work is carried by the proxy approximation, which relies on DINOv2's robustness to degradations, and by a learned projection MLP g_phi that maps diffusion features into the DINOv2 embedding space. The theory identifies REPA as an MMD surrogate and as a local contraction operator on latent states.
What would settle it
Measure the proxy approximation error ApproxErr(x, bar x) = E_n ||f_DINOv2(x) - f_DINOv2(bar x)||^2 on box inpainting and motion deblurring, and check whether REPA's LPIPS/FID gains correlate with it; if a degradation with high proxy error, such as severe motion blur, yields worse LPIPS than the baseline, the central claim is refuted.
Extended reading notes
Core claim
At inference time, the paper adds a regularizer, termed REPA, to the reverse sampling loop of latent diffusion and flow models. For each patch, REPA computes the cosine similarity between the diffusion model's intermediate hidden tokens, projected by a learned MLP, and DINOv2 patch features of a proxy image, and turns that similarity into a gradient step on the latent state. The central empirical discovery is that this alignment improves perceptual metrics, lower LPIPS and lower FID, on four inverse tasks and two datasets, while maintaining or slightly improving distortion metrics, and that it can reach the same LPIPS as the unregularized solver with substantially fewer steps. The paper's th
Load-bearing premise
The whole method rests on the assumption that DINOv2 features of a proxy, the corrupted measurement or the model's current estimate, are close to DINOv2 features of the true clean image; if that approximation fails, the regularizer pulls the reconstruction toward the wrong features and can worsen quality.
Editorial extensions
If this is right
- If the claim holds, any latent diffusion or flow inverse solver can be upgraded by adding one gradient term, provided the diffusion model has a trained REPA projection head.
- Fewer discretization steps are needed for the same perceptual quality, directly reducing sampling cost.
- Perceptual metrics (LPIPS and FID) improve across all four tested inverse problems, while PSNR and SSIM are not sacrificed.
- The method extends to flow-based models and to independently pretrained models after finetuning with the REPA objective.
- The theory implies a principled link between feature-space divergence minimization and inverse problem reconstruction.
Reading between the lines
- Beyond the paper, the proxy approximation is only validated for super-resolution and Gaussian deblurring; box inpainting and motion deblurring are not measured, so a natural test is whether REPA's gains survive when DINOv2 features are not robust to the specific degradation.
- Using the model's own denoised estimate as the proxy risks reinforcing its own errors: if the estimate is wrong, REPA will sharpen the wrong features, effectively anchoring hallucinations. A comparison of a frozen measurement proxy against an adaptive proxy on severe degradations would reveal this failure mode.
- REPA might transfer beyond DINOv2: any self-supervised encoder with similar invariance properties could serve as the alignment target, and the MMD interpretation suggests the benefit scales with the encoder's perceptual relevance.
- The efficiency claim suggests a practical recipe for real-time restoration: fewer steps plus a cheap regularizer could make high-quality diffusion-based inverse solving viable on lower-budget hardware.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes an inference-time regularizer for latent diffusion/flow-based inverse-problem solvers: at each reverse step, in addition to the measurement-consistency update, the algorithm adds a gradient that maximizes cosine similarity between DINOv2 features of a proxy image and the diffusion model's internal features after a fixed number of transformer blocks, mapped through a pretrained MLP. Because ground-truth images are unavailable at inference, the proxy is initialized from the measurement and later replaced by the model's current denoised estimate. The paper presents two theoretical results: Proposition 4.3 connects the REPA objective to a feature-space discrepancy, and Proposition 4.4 claims that under Jacobian regularity assumptions a REPA update locally contracts the latent state toward the clean image's latent state. Experiments on ImageNet and FFHQ cover 4x super-resolution, box inpainting, Gaussian deblurring, and motion deblurring, comparing Latent DPS and ReSample with and without REPA, plus DPS and Latent DAPS as external baselines. The paper reports consistent LPIPS/FID improvements, comparable PSNR/SSIM, and 2-4x reductions in the number of discretization steps.
Significance. The idea is timely and practically appealing: a simple, plug-in regularizer that improves perceptual quality and reduces sampling steps would be a useful contribution to the inverse-problem literature. The Latent DPS with/without REPA comparison is a clean same-backbone ablation and is the strongest evidence in the paper. The appendix contains complete proofs of both propositions, which is a strength. However, the generality claim is weakened by a confounded ReSample comparison, the absence of error bars on the reported metrics, and incomplete validation of the central proxy-approximation assumption. The theoretical contraction result also has internal inconsistencies that need correction. If these issues are addressed, the paper could make a solid contribution.
major comments (4)
- [§5.1, Table 1; Appendix A.3] The ReSample rows do not provide a controlled ablation. Appendix A.3 explicitly states that the authors' ReSample adaptation omits the pixel-space stage of the original three-stage algorithm, and that this omission, together with REPA, yields sharper results. The baseline in Table 1, however, is the original ReSample algorithm. Therefore the gains in the 'Resample + REPA' rows may be caused by the solver modification (removing the pixel-space stage) rather than by REPA. Report the modified solver without REPA (same omission, same inner-loop settings) as the baseline; without this, the claim that REPA improves a second state-of-the-art solver is not established.
- [§5, Table 1] All metrics are averaged over 100 validation images and no error bars, standard deviations, or statistical significance tests are reported. FID on 100 images is high-variance; several reported gains are small (e.g., FFHQ Gaussian deblur LPIPS 0.192 vs 0.186; FFHQ motion deblur PSNR 27.20 vs 27.16). Appendix A.6 says PSNR/SSIM are computed on the average of five runs, but it is not clear this applies to LPIPS/FID. Please provide confidence intervals or multiple-seed results for at least the main Latent DPS comparisons.
- [Prop. 4.4 / Appendix A.1 (Prop. A.3)] The proof of the contraction result has mismatched constants. The proposition defines C1 = 1 - λ m_G m_phi, while the proof obtains a factor 1 - 2λ m_G^2 m_phi^2 after inequality (21); the stated radius condition only makes the coefficient strictly less than 1 without a uniform gap, so the claimed C1 ∈ (0,1) is not established. The additive constant C2 is defined with a factor 1/√N in the proposition but the proof concludes C = 2λ M_G M_phi without it. Also, Assumptions A.2.2-A.2.3 (uniform positive lower bounds on J^T J) are not verified for the actual diffusion encoder and projection head. Please correct the constants and add a discussion of whether/why these regularity assumptions can be expected to hold.
- [Eq. (7), §3; Appendix A.4] The method's central premise is that DINOv2 features of the proxy (measurement or current denoised estimate) approximate those of the clean image. Appendix A.4 quantifies this only for super-resolution and Gaussian deblurring; box inpainting and motion deblurring are not measured. For motion deblurring, the blur is severe and the measurement-based proxy may violate Eq. (7); for box inpainting the proxy is a mask-weighted combination with the model's own estimate (Appendix A.5), which can reinforce reconstruction errors. Please extend the robustness analysis to all four tasks and discuss what happens when Eq. (7) is violated; this is load-bearing for the 'consistent improvements across tasks' claim.
minor comments (6)
- [Abstract] The sentence 'we demonstrate the generality of our approach by We integrate REPA into multiple state-of-the-art inverse problem solvers' is incomplete; fix the grammar.
- [Algorithm 1, line 9] The proxy update uses 'x_t' in a latent-space algorithm: should be 'z_t'. Also, the notation E[z0 | x_t] overloads the encoder E with conditional expectation; please disambiguate.
- [Algorithms 2 and 3] Same notation issue: E is used both for the VAE encoder and for the conditional expectation E[z0 | z_t]. Consider using a different symbol or an explicit estimator name.
- [Prop. 4.3] The quantity named MMD is only a mean-embedding (first-moment) discrepancy, not the full maximum mean discrepancy with a kernel. Rename it or justify the terminology.
- [Conclusions / Appendix A.7] Typos: 'clen latent state' should be 'clean latent state'; 're alignment' should be 'representation alignment'; 'trough the lens' should be 'through the lens'.
- [Table 3] Heading 'Resampling + REPA' is inconsistent with 'ReSample + REPA' used elsewhere; keep terminology uniform.
Circularity Check
No significant circularity: the theoretical results are conditional bounds and the main claims are empirical comparisons against external baselines.
full rationale
The paper's derivation chain is not circular by construction. The core approximation in Eq. (7), cproxy ≡ f_DINOv2(¯x0) ≈ f_DINOv2(x0), is an explicit assumption that is empirically probed in Appendix A.4; it is not defined so as to guarantee the conclusion, and the paper does not equate the regularizer's success with this assumption. Proposition 4.3 is a mathematical inequality relating expected REPA to an MMD in DINOv2 feature space plus ApproxErr and MisREPA terms; it follows from the definitions and does not assume that REPA improves inverse-problem solvers. Proposition 4.4 is explicitly local and conditional: it proves a contraction only under Assumption A.2 and with error terms ApproxErr and MisREPA appearing additively, so it does not reduce the claim 'REPA improves reconstruction' to its own premise. The REPA and DINOv2 components are taken from prior work by other groups (Yu et al., Oquab et al.); self-citations (e.g., Daras et al. survey, Rout et al.) are contextual and not load-bearing. The ReSample comparison in Table 1 is confounded because Appendix A.3 states the authors' adaptation 'omits the pixel-space stage,' so 'Resample + REPA' differs from the baseline in more than the regularizer; this is an experimental-control/correctness concern, not a circularity. Similarly, Appendix A.4 validates proxy robustness only for super-resolution and Gaussian deblurring, leaving inpainting and motion deblurring unmeasured; that is a support gap, not a circular derivation. Under the rule that circularity requires a specific reduction by construction, no such step is present.
Assumptions & free parameters
free parameters (4)
- λ (REPA regularizer strength) =
varies: 0.01–0.05 (ImageNet) and 0.5–3.25 (FFHQ) for Latent DPS; different values for ReSample (Tables 2–3)
- κ (measurement-consistency step size) =
varies per task and dataset (Tables 2–3)
- t_cutoff (proxy update schedule) =
FFHQ: switch after 80% of steps; ImageNet: measurement-based proxy throughout
- Resample inner-loop iterations and learning rate =
max iters 100–300; inner lr 0.0001–0.005 (Table 3)
assumptions (4)
- domain assumption DINOv2 feature embeddings are ℓ2-normalized (Prop 4.3 assumption)
- domain assumption DINOv2 representations are robust to the studied degradations, so ApproxErr is small
- ad hoc to paper Regularity assumptions A.2.1–A.2.4 on the Jacobians of DiffEnc and g_phi (Lipschitz continuity, positive lower singular-value bounds) hold along the trajectory
- domain assumption The diffusion model's internal representations are already aligned with DINOv2 (MisREPA small), either from REPA pretraining or finetuning
Cite this review
Pith. "Pith review of Align & Invert: Solving Inverse Problems with Diffusion and Flow-based Models via Representation Alignment." pith.science (2026). https://pith.science/paper/AJD6UNTY
@misc{pith2026251116870,
author = {Pith},
title = {Pith review of: Align & Invert: Solving Inverse Problems with Diffusion and Flow-based Models via Representation Alignment},
year = {2026},
howpublished = {\url{https://pith.science/paper/AJD6UNTY}},
note = {Machine review of arXiv:2511.16870}
}
read the original abstract
Enforcing alignment between the internal representations of diffusion or flow-based generative models and those of pretrained self-supervised encoders has recently been shown to provide a powerful inductive bias, improving both convergence and sample quality. In this work, we extend this idea to inverse problems, where pretrained generative models are employed as priors. We propose applying representation alignment (REPA) between diffusion or flow-based models and a DINOv2 visual encoder, to guide the reconstruction process at inference time. Although ground-truth signals are unavailable in inverse problems, we empirically show that aligning model representations of approximate target features can substantially enhance reconstruction quality and perceptual realism. We provide theoretical results showing (a) that REPA regularization can be viewed as a variational approach for minimizing a divergence measure in the DINOv2 embedding space, and (b) how under certain regularity assumptions REPA updates steer the latent diffusion states toward those of the clean image. These results offer insights into the role of REPA in improving perceptual fidelity. Finally, we demonstrate the generality of our approach by We integrate REPA into multiple state-of-the-art inverse problem solvers, and provide extensive experiments on super-resolution, box inpainting, Gaussian deblurring, and motion deblurring confirming that our method consistently improves reconstruction quality, while also providing efficiency gains reducing the number of required discretization steps.
Figures
Figures from the paper (12 more)
Reference graph
Works this paper leans on
-
[1]
Albergo, M. S., Boffi, N. M., and Vanden-Eijnden, E. Stochastic interpolants: A unifying framework for flows and diffusions.arXiv preprint arXiv:2303.08797,
-
[2]
ImageNet uses Gaussian noiseσ= 0.01, while FFHQ usesσ= 0.05
15 Align & Invert: Solving Inverse Problems with Diffusion and Flow-based Models via Representation Alignment Table 2.Hyperparameters used for Latent DPS + REPA. ImageNet uses Gaussian noiseσ= 0.01, while FFHQ usesσ= 0.05. ImageNet FFHQ Taskκ λ κ λ Super-resolution 2.0 0.010.025 1.25 Gaussian deblurring 0.25 0.050.05 0.5 Motion deblurring 0.5 0.010.05 0.5...
2024
-
[3]
In addition to the parameters κ and λ, which play analogous roles to those in Latent DPS, we also tune the maximum number of inner-loop optimization steps and the corresponding inner learning rate of the Resample procedure (see Algorithm 3). A.4. Robustness of DINOv2 Representations to Corruptions To evaluate the robustness of DINOv2 representations under...
2023
-
[6]
T., Ben-Hamu, H., Nickel, M., and Le, M
Lipman, Y ., Chen, R. T., Ben-Hamu, H., Nickel, M., and Le, M. Flow matching for generative modeling. In11th International Conference on Learning Representations, ICLR 2023,
2023
-
[7]
Oquab, M., Darcet, T., Moutakanni, T., V o, H., Szafraniec, M., Khalidov, V ., Fernandez, P., Haziza, D., Massa, F., El- Nouby, A., et al. Dinov2: Learning robust visual features 9 Align & Invert: Solving Inverse Problems with Diffusion and Flow-based Models via Representation Alignment without supervision.arXiv preprint arXiv:2304.07193,
-
[8]
It- erative tilting for diffusion fine-tuning.arXiv preprint arXiv:2512.03234,
Pachebat, J., Conforti, G., Durmus, A., and Janati, Y . It- erative tilting for diffusion fine-tuning.arXiv preprint arXiv:2512.03234,
-
[9]
Patel, M., Wen, S., Metaxas, D. N., and Yang, Y . Steering rectified flow models in the vector field for controlled im- age generation.arXiv preprint arXiv:2412.00100,
-
[10]
Knn-diffusion: Im- age generation via large-scale retrieval.arXiv preprint arXiv:2204.02849,
Sheynin, S., Ashual, O., Polyak, A., Singer, U., Gafni, O., Nachmani, E., and Taigman, Y . Knn-diffusion: Im- age generation via large-scale retrieval.arXiv preprint arXiv:2204.02849,
Show all 18 references
-
[12]
U-repa: Aligning diffusion u-nets to vits.arXiv preprint arXiv:2503.18414,
Tian, Y ., Chen, H., Zheng, M., Liang, Y ., Xu, C., and Wang, Y . U-repa: Aligning diffusion u-nets to vits.arXiv preprint arXiv:2503.18414,
-
[13]
Learning diffusion models with flexible representation guidance.arXiv preprint arXiv:2507.08980, 2025a
Wang, C., Zhou, C., Gupta, S., Lin, Z., Jegelka, S., Bates, S., and Jaakkola, T. Learning diffusion models with flexible representation guidance.arXiv preprint arXiv:2507.08980, 2025a. Wang, Z., Bovik, A. C., Sheikh, H. R., and Simoncelli, E. P. Image quality assessment: from ...
-
[15]
Exploring representation-aligned latent space for better generation
Xu, W., Yue, X., Wang, Z., Teng, Y ., Zhang, W., Liu, X., Zhou, L., Ouyang, W., and Bai, L. Exploring representation-aligned latent space for better generation. arXiv preprint arXiv:2502.00359,
-
[16]
Representation alignment for generation: Training diffusion transformers is easier than you think
Yu, S., Kwak, S., Jang, H., Jeong, J., Huang, J., Shin, J., and Xie, S. Representation alignment for generation: Training diffusion transformers is easier than you think. arXiv preprint arXiv:2410.06940,
-
[2004]
Repa works until it doesn’t: Early-stopped, holistic align- ment supercharges diffusion training.arXiv preprint arXiv:2505.16792, 2025b
Wang, Z., Zhao, W., Zhou, Y ., Li, Z., Liang, Z., Shi, M., Zhao, X., Zhou, P., Zhang, K., Wang, Z., et al. Repa works until it doesn’t: Early-stopped, holistic align- ment supercharges diffusion training.arXiv preprint arXiv:2505.16792, 2025b. Xiang, W., Yang, H., Huang, D., a...
-
[2019]
The principles of diffusion models.arXiv preprint arXiv:2510.21890,
Lai, C.-H., Song, Y ., Kim, D., Mitsufuji, Y ., and Ermon, S. The principles of diffusion models.arXiv preprint arXiv:2510.21890,
-
[2022]
V ., Seitzer, M., Baldassarre, F., Oquab, M., Jose, C., Khalidov, V ., Szafraniec, M., Yi, S., Ramamonjisoa, M., et al
Sim´eoni, O., V o, H. V ., Seitzer, M., Baldassarre, F., Oquab, M., Jose, C., Khalidov, V ., Szafraniec, M., Yi, S., Ramamonjisoa, M., et al. Dinov3.arXiv preprint arXiv:2508.10104,
-
[2023]
C., Milanfar, P., Dimakis, A
Daras, G., Chung, H., Lai, C.-H., Mitsufuji, Y ., Ye, J. C., Milanfar, P., Dimakis, A. G., and Delbracio, M. A survey on diffusion models for inverse problems.arXiv preprint arXiv:2410.00083,
-
[2024]
Dreamsim: Learning new dimensions of human visual similarity using synthetic data.arXiv preprint arXiv:2306.09344,
Fu, S., Tamir, N., Sundaram, S., Chai, L., Zhang, R., Dekel, T., and Isola, P. Dreamsim: Learning new dimensions of human visual similarity using synthetic data.arXiv preprint arXiv:2306.09344,
-
[2025]
Repa-e: Unlocking vae for end-to-end tuning with latent diffusion transformers.arXiv preprint arXiv:2504.10483,
Leng, X., Singh, J., Hou, Y ., Xing, Z., Xie, S., and Zheng, L. Repa-e: Unlocking vae for end-to-end tuning with latent diffusion transformers.arXiv preprint arXiv:2504.10483,
Reviewed August 3, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.