REVIEW 3 major objections 4 minor 40 references
Stabilizing RED using the Koopman Operator
T0 review · 3 major / 4 minor · reviewed 2026-08-05 · deepseek-v4-flash
Pith's one-line read Adaptive Koopman monitor stops RED reconstruction from diverging
desk verdict A solid empirical letter: the Koopman/DMD step-size controller is genuinely new for RED, the experiments are consistent, and the main weakness is that the stability proxy is heuristic rather than proven. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The Koopman operator K_t, approximated in a 66-dimensional observable space by ordinary least squares from a window of recent RED iterates. The paper uses the spectral radius of K_t as a proxy for the local contraction rate of the full RED iteration in R^n; values above 1 trigger an exponential step-size reduction. The feature map (global statistics, 4x4 pooling, low-frequency DCT) is the handcrafted observable choice that makes the low-dimensional fit informative.
What would settle it
Construct or find a pretrained denoiser and task where, at the moment vanilla RED begins to diverge, rho(K_t) remains below 1, so SKOOP-RED never shrinks gamma; if PSNR then drops while the monitor reports stability, the spectral-radius criterion is falsified. Conversely, if rho(K_t) exceeds 1 during runs that remain stable, the controller is over-conservative.
Extended reading notes
Core claim
SKOOP-RED is an adaptive step-size schedule for RED. At checkpoints, the most recent iterates are projected into a 66-dimensional feature space combining per-channel mean and standard deviation, 4x4 block means, and low-frequency DCT coefficients. A linear operator K_t is fitted by least squares to advance these features one step, approximating the Koopman operator of the RED dynamics; its spectral radius rho(K_t) is read as a local stability indicator. If rho(K_t) >= 1, the step size gamma is multiplied by exp(-beta(rho-1)) with beta = 2; otherwise gamma is unchanged. The paper reports that this simple rule produces stable, high-PSNR reconstructions where vanilla RED diverges, across DnCNN,
Load-bearing premise
The result rests on the assumption that the spectral radius of a 66-dimensional least-squares linear operator fitted to recent RED iterates in a handcrafted feature space is a faithful early warning of whether the full RED iteration in image space is about to diverge.
Editorial extensions
If this is right
- RED can be run with arbitrary pretrained deep denoisers without imposing nonexpansiveness or retraining.
- The adaptive step size removes the need for early stopping at an unpredictable breakdown point.
- The same denoiser and forward model can be used across tasks; only the Koopman monitor needs to be fitted online.
- The overhead is modest enough for iterative restoration in practice.
- The approach is not tied to gradient descent structure and could be transferred to other RED variants.
Reading between the lines
- The stability indicator could be sharpened by comparing rho(K_t) against the spectral radius of the full Jacobian on a few examples; if they disagree, a richer observable set would be needed.
- A more principled choice of the feature map and window, rather than sensitivity-based selection, might yield a bound connecting the finite-dimensional Koopman spectrum to the true RED dynamics.
- The same checkpointed Koopman monitor may generalize to other iterative reconstruction algorithms, such as plug-and-play ADMM or diffusion-based restoration, where divergence also appears after many iterations.
- Since the controller only shrinks the step size, it cannot fix a denoiser that is unstable in every direction; the method's ceiling is the quality of the underlying denoiser.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes SKOOP-RED, an adaptive step-size controller for Regularization-by-Denoising (RED) image reconstruction. At checkpoints during the RED iteration, the method projects recent iterates into a 66-dimensional handcrafted feature space, fits a low-dimensional Koopman operator K_t by least squares (Eq. 7), and shrinks the RED step size whenever the spectral radius ρ(K_t) exceeds 1 (Algorithm 1). The authors claim this stabilizes RED with black-box deep denoisers, avoids retraining, adds less than 20% runtime overhead, and outperforms Vanilla RED and Equivariant RED across deblurring and superresolution tasks with DnCNN, DRUNet, GS-DRUNet, and DiffUNet. Code is provided.
Significance. If the empirical claims hold, this is a practically useful contribution: RED with modern deep denoisers is known to be unstable, and a model-agnostic, low-overhead stabilization mechanism would strengthen the case for RED in real imaging systems. The paper is clearly written, the experiments cover four denoisers and three tasks, and the code release is a concrete asset. However, the central mechanism—using the spectral radius of a 66-dimensional OLS-fitted Koopman operator as a proxy for the stability of the full RED dynamics—is asserted without a supporting bound or direct validation. The evaluation also tunes key hyperparameters on the same test tasks and reports no variance over the 15-image set. These gaps are load-bearing for the claim of 'consistent stability improvements,' but they are addressable with additional analysis and a more rigorous evaluation protocol.
major comments (3)
- [Sec. 2.3, Eqs. (7)-(8)] The controller's decision rule assumes that ρ(K_t) ≥ 1 indicates potential instability of the full RED map T_γ in R^n. This link is not derived. K_t is an OLS fit in a low-dimensional feature space; without an error bound relating the approximate spectrum to the spectral properties of DT_γ, the feature map can miss genuinely unstable modes (false negatives, no step-size reduction) or introduce spurious expanding modes (false positives, unnecessary slowdown). The paper's own appendix (Section A) describes the feature map as 'empirically' useful, which effectively concedes that this link is heuristic. To make the central claim load-bearing, the authors should provide either a formal error estimate in terms of the feature-map invariance/closure and the residual of Eq. (7), or direct validation on small-scale problems comparing ρ(K_t) against the spectral radius of the full Jacobian along th
- [Sec. 3, Tables 2 and S1; Sec. 2.4] The stability claim rests on 15 test images (Set15C), and Tables 2 and S1 report only mean PSNR values with no variance, percentiles, or per-image breakdowns. Equally important, the feature map, window size w, stride r, and decay rate β are selected via sensitivity studies on the same tasks (Section 2.4, Fig. S2), creating a risk of indirect overfitting to the evaluation set. I request: (i) report standard deviations or per-image results; (ii) freeze all SKOOP parameters on a development split and evaluate on held-out tasks, kernel types, or denoisers; (iii) include an ablation that replaces ρ(K_t) with a simple heuristic (e.g., residual-norm growth or a fixed decay schedule) to demonstrate that the Koopman spectral-radius monitoring, rather than mere step-size adaptation, is what provides the stability benefit.
- [Algorithm 1 and Eq. (8)] The control law only shrinks γ and never increases it. This is not necessarily wrong, but it means any false positive permanently slows the iteration, and the paper does not discuss recovery from overly conservative step sizes. Moreover, the resulting non-autonomous iteration is not analyzed at all; the only theoretical framing is the heuristic spectral-radius test. A formal statement for a simplified or linearized model (e.g., under what conditions the adaptive scheme keeps γ bounded away from zero and prevents divergence) would substantially strengthen the paper and clarify what SKOOP-RED can guarantee.
minor comments (4)
- [Sec. 2.3, Eq. (6) vs. Eq. (7)] Equation (6) states K_t ψ(x_τ) = ψ(x_{τ+1}) for t−w+1 ≤ τ ≤ t, but for τ = t this would require the future iterate x_{t+1}. Equation (7) correctly sums only up to τ = t−1. Please fix the index range in Eq. (6) to avoid confusion.
- [Sec. 3, Table 1] The text says SKOOP-RED adds 'only a small 15–20% runtime overhead per iteration,' but Table 1 reports overheads of 10.97–14.11%, and Section 2.4 reports 11% for DiffUNet. Please reconcile these numbers.
- [Table 2] The abbreviation 'Equiv.RED' is used without a definition in the table caption; it appears to refer to equivariant RED from [18], but this should be stated explicitly.
- [Sec. 2.4] The paper says K_t is computed using the technique in [33] (DMD), but Eq. (7) is a least-squares fit. Please specify which DMD variant is used (e.g., exact DMD, projected DMD) and what rank/regularization, if any, is applied.
Circularity Check
No significant circularity: SKOOP-RED is a feedback controller; no target outcome is encoded in the fitted Koopman operator.
full rationale
The paper's derivation chain is a feedback-control mechanism: at checkpoints, K_t is fit by OLS to the recent RED trajectory (Eq. 7), its spectral radius is computed, and if rho(K_t)>1 the step size is shrunk via Eq. (8). This is not circular because the fitted operator is estimated from past iterates before the step-size correction is applied, and it does not encode the target PSNR or stability outcome. The claim that rho(K_t)>=1 indicates potential instability is an empirical modeling assumption—an unproven proxy, which is a correctness/robustness risk—not a result equivalent to the input by construction. No parameter is fitted to the reported final PSNR and then renamed a prediction; the reported PSNR is an outcome of the adaptive policy, and the paper tests this out-of-sample on multiple denoisers, tasks, and kernels. The self-citations ([19], [24], [25]) concern different denoiser properties and are not load-bearing for the Koopman mechanism. The supplementary's sensitivity studies for window/stride/beta are hyperparameter tuning, not circular derivation. Therefore no specific reduction of the kind required for a circularity finding can be exhibited.
Assumptions & free parameters
free parameters (5)
- decay rate beta =
2
- window size w and checkpoint stride r =
w=30, r=10
- feature map design (d=66, 4x4 pooling, DCT coefficients) =
d=66
- spectral-radius trigger =
1
- RED step size gamma_0 and regularization lambda =
not reported
assumptions (5)
- standard math The Koopman operator on a Hilbert space of observables is linear and its spectrum characterizes the stability of the underlying map.
- standard math DMD/least-squares regression approximates the Koopman operator from trajectory data.
- ad hoc to paper The 66-dimensional handcrafted feature map preserves the stability-relevant information of the full RED iteration.
- ad hoc to paper The spectral radius of the fitted finite-dimensional K_t is a valid indicator of instability of the full RED dynamics.
- domain assumption The recent trajectory window is representative of future local dynamics.
Cite this review
Pith. "Pith review of Stabilizing RED using the Koopman Operator." pith.science (2026). https://pith.science/paper/R3BOXZY5
@misc{pith2026250905736,
author = {Pith},
title = {Pith review of: Stabilizing RED using the Koopman Operator},
year = {2026},
howpublished = {\url{https://pith.science/paper/R3BOXZY5}},
note = {Machine review of arXiv:2509.05736}
}
read the original abstract
The widely used RED (Regularization-by-Denoising) framework uses pretrained denoisers as implicit regularizers for model-based reconstruction. Although RED generally yields high-fidelity reconstructions, the use of black-box denoisers can sometimes lead to instability. In this letter, we propose a data-driven mechanism to stabilize RED using the Koopman operator, a classical tool for analyzing dynamical systems. Specifically, we use the operator to capture the local dynamics of RED in a low-dimensional feature space, and its spectral radius is used to detect instability and formulate an adaptive step-size rule that is model-agnostic, has modest overhead, and requires no retraining. We test this with several pretrained denoisers to demonstrate the effectiveness of the proposed Koopman stabilization.
Figures
Figures from the paper (3 more)
Reference graph
Works this paper leans on
-
[1]
Charles A Bouman.Foundations of Computational Imaging: A Model-Based Approach. SIAM, 2022
work page 2022
-
[2]
Nonlinear total variation based noise removal algorithms.Physica D: Nonlinear Phenomena, 60(1-4):259–268, 1992
Leonid I Rudin, Stanley Osher, and Emad Fatemi. Nonlinear total variation based noise removal algorithms.Physica D: Nonlinear Phenomena, 60(1-4):259–268, 1992
1992
-
[3]
Markov Random Field image models and their applica- tions to computer vision.Proc
Stuart Geman and Christine Graffigne. Markov Random Field image models and their applica- tions to computer vision.Proc. International Congress of Mathematicians, 1:2, 1986
work page 1986
-
[4]
Vese and Carole Le Guyader.Variational Methods in Image Processing
Luminita A. Vese and Carole Le Guyader.Variational Methods in Image Processing. CRC Press, 2016. 7
work page 2016
-
[5]
Learning a deep convolutional network for image super-resolution.Proc
Chao Dong, Chen Change Loy, Kaiming He, and Xiaoou Tang. Learning a deep convolutional network for image super-resolution.Proc. ECCV, pages 184–199, 2014
work page 2014
-
[6]
MambaIR: A simple baseline for image restoration with state-space model.Proc
Hang Guo, Jinmin Li, Tao Dai, Zhihao Ouyang, Xudong Ren, and Shu-Tao Xia. MambaIR: A simple baseline for image restoration with state-space model.Proc. ECCV, pages 222–241, 2025
work page 2025
-
[7]
SwinIR: Image restoration using swin transformer.Proc
Jingyun Liang, Jiezhang Cao, Guolei Sun, Kai Zhang, Luc Van Gool, and Radu Timofte. SwinIR: Image restoration using swin transformer.Proc. ICCV Workshops, pages 1833–1844, 2021
work page 2021
-
[8]
Restormer: Efficient transformer for high-resolution image restoration.Proc
Syed Waqas Zamir, Aditya Arora, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, and Ming-Hsuan Yang. Restormer: Efficient transformer for high-resolution image restoration.Proc. CVPR, pages 5728–5739, 2022
work page 2022
Show all 40 references
-
[9]
Denoising diffusion restoration models.Proc
Bahjat Kawar, Michael Elad, Stefano Ermon, and Jiaming Song. Denoising diffusion restoration models.Proc. NeurIPS, pages 23593–23606, 2022
2022
-
[10]
Denoising diffusion models for plug-and-play image restoration.Proc
Yuanzhi Zhu, Kai Zhang, Jingyun Liang, Jiezhang Cao, Bihan Wen, Radu Timofte, and Luc Van Gool. Denoising diffusion models for plug-and-play image restoration.Proc. CVPR Workshops, pages 1219–1229, 2023
2023
-
[11]
Sreehari, S
S. Sreehari, S. V . Venkatakrishnan, B. Wohlberg, G. T. Buzzard, L. F. Drummy, J. P . Simmons, and C. A. Bouman. Plug-and-play priors for bright field electron tomography and sparse interpolation.IEEE T rans. Comput. Imaging, 2(4):408–423, 2016
2016
-
[12]
Gradient step denoiser for convergent plug-and-play.Proc
Samuel Hurault, Arthur Leclaire, and Nicolas Papadakis. Gradient step denoiser for convergent plug-and-play.Proc. ICLR, 2022
2022
-
[13]
The little engine that could: Regularization by Denoising (RED).SIAM J
Yaniv Romano, Michael Elad, and Peyman Milanfar. The little engine that could: Regularization by Denoising (RED).SIAM J. Imaging Sci., 10(4):1804–1844, 2017
2017
-
[14]
Regularization by denoising: Clarifications and new interpretations.IEEE T rans
Edward T Reehorst and Philip Schniter. Regularization by denoising: Clarifications and new interpretations.IEEE T rans. Comput. Imaging, 5(1):52–67, 2018
2018
-
[15]
Zhang, Y
K. Zhang, Y. Li, W. Zuo, L. Zhang, L. Van Gool, and R. Timofte. Plug-and-play image restoration with deep denoiser prior.IEEE T rans. Pattern Anal. Mach. Intell., 44(10):6360–6376, 2021
2021
-
[16]
Plug-and-play algorithms for large-scale snapshot compressive imaging.Proc
Xin Yuan, Yang Liu, Jinli Suo, and Qionghai Dai. Plug-and-play algorithms for large-scale snapshot compressive imaging.Proc. CVPR, pages 1447–1457, 2020
2020
-
[17]
Cohen, M
R. Cohen, M. Elad, and P . Milanfar. Regularization by denoising via fixed-point projection (RED-PRO).SIAM J. Imaging Sci., 14(3):1374–1406, 2021
2021
-
[18]
Terris, T
M. Terris, T. Moreau, N. Pustelnik, and J. Tachella. Equivariant plug-and-play image recon- struction.Proc. CVPR, pages 25255–25264, 2024
2024
-
[19]
Averaged deep denoisers for image regularization.J
Pravin Nair and Kunal N Chaudhury. Averaged deep denoisers for image regularization.J. Math. Imaging Vis., pages 1–18, 2024
2024
-
[20]
Pesquet, A
J.-C. Pesquet, A. Repetti, M. Terris, and Y. Wiaux. Learning maximally monotone operators for image recovery.SIAM J. Imaging Sci., 14(3):1206–1237, 2021
2021
-
[21]
Convolutional proximal neural networks and plug-and-play algorithms.Linear Algebra Appl., 631:203–234, 2021
Johannes Hertrich, Sebastian Neumayer, and Gabriele Steidl. Convolutional proximal neural networks and plug-and-play algorithms.Linear Algebra Appl., 631:203–234, 2021
2021
-
[22]
Learning weakly convex regularizers for convergent image-reconstruction algorithms.SIAM J
Alexis Goujon, Sebastian Neumayer, and Michael Unser. Learning weakly convex regularizers for convergent image-reconstruction algorithms.SIAM J. Imaging Sci., 17(1):91–115, 2024
2024
-
[23]
E. Ryu, J. Liu, S. Wang, X. Chen, Z. Wang, and W. Yin. Plug-and-play methods provably converge with properly trained denoisers.Proc. ICML, pages 5546–5557, 2019
2019
-
[24]
Nair and K
P . Nair and K. N. Chaudhury. Convergent plug-and-play using contractive denoisers.Proc. ICASSP, 2023
2023
-
[25]
Kumar and K
B. Kumar and K. N. Chaudhury. Lipschitz-constrained convolutional layers using convex projection.Proc. ICASSP, 2023. 8
2023
-
[26]
Levin, Y
A. Levin, Y. Weiss, F. Durand, and W. T. Freeman. Understanding and evaluating blind deconvolution algorithms.Proc. CVPR, pages 1964–1971, 2009
1964
-
[27]
B. O. Koopman. Hamiltonian systems and transformation in Hilbert space.Proc. Natl. Acad. Sci., 17(5):315–318, 1931
1931
-
[28]
Spectral properties of dynamical systems, model reduction and decompositions
Igor Mezi´c. Spectral properties of dynamical systems, model reduction and decompositions. Nonlinear Dynamics, 41:309–325, 2005
2005
-
[29]
C. W. Rowley, I. Mezi´c, S. Bagheri, P . Schlatter, and D. S. Henningson. Spectral analysis of nonlinear flows.J. Fluid Mech., 641:115–127, 2009
2009
-
[30]
S. L. Brunton, B. W. Brunton, J. L. Proctor, E. Kaiser, and J. N. Kutz. Chaos as an intermittently forced linear system.Nat. Commun., 8(1):19, 2017
2017
-
[31]
Proof of the quasi-ergodic hypothesis.Proc
John von Neumann. Proof of the quasi-ergodic hypothesis.Proc. Natl. Acad. Sci., 18(1):70–82, 1932
1932
-
[32]
Comparison of systems with complex behavior.Physica D: Nonlinear Phenomena, 197(1-2):101–133, 2004
Igor Mezi´c and Andrzej Banaszuk. Comparison of systems with complex behavior.Physica D: Nonlinear Phenomena, 197(1-2):101–133, 2004
2004
-
[33]
P . J. Schmid. Dynamic mode decomposition of numerical and experimental data.J. Fluid Mech., 656:5–28, 2010
2010
-
[34]
J. H. Tu.Dynamic Mode Decomposition: Theory and Applications. Ph.d. dissertation, Princeton University, 2013
2013
-
[35]
Korda and I
M. Korda and I. Mezi ´c. On convergence of extended dynamic mode decomposition to the Koopman operator.J. Nonlinear Sci., 28(2):687–710, 2017
2017
-
[36]
Zhang, W
K. Zhang, W. Zuo, Y. Chen, D. Meng, and L. Zhang. Beyond a Gaussian denoiser: Residual learning of deep CNN for image denoising.IEEE T rans. Image Process., 26(7):3142–3155, 2017
2017
-
[37]
J. Ho, A. Jain, and P . Abbeel. Denoising diffusion probabilistic models.Proc. NeurIPS, 33:6840– 6851, 2020
2020
-
[38]
Deepinverse: A Python package for solving imaging inverse problems with deep learning.arXiv preprint arXiv:2505.20160, 2025
Juli´an Tachella, Matthieu Terris, Samuel Hurault, Andrew Wang, Dongdong Chen, Minh-Hai Nguyen, Maxime Song, Thomas Davies, Leo Davy, Jonathan Dong, et al. Deepinverse: A Python package for solving imaging inverse problems with deep learning.arXiv preprint arXiv:2505.20160, 2025
2025 arXiv
-
[39]
The little engine that could: Regularization by Denoising (RED) [code].https://github.com/google/RED, 2017
Yaniv Romano, Michael Elad, and Peyman Milanfar. The little engine that could: Regularization by Denoising (RED) [code].https://github.com/google/RED, 2017
2017
-
[40]
Regularization by denoising: Clarifications and new interpretations [code].https://github.com/edward-reehorst/On_RED, 2018
Edward T Reehorst and Philip Schniter. Regularization by denoising: Clarifications and new interpretations [code].https://github.com/edward-reehorst/On_RED, 2018. 9 A Supplementary Material 0 2000 4000 6000 8000 10000 20.0 22.5 25.0 27.5 30.0 Vanilla RED SKOOP-RED 0 2000 4000 ...
2018
Reviewed August 5, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.