{"id":"230c6894-39b0-43f0-b8c7-caaf51cabbb9","arxiv_id":"2506.13239","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"ReTune proves that restarting a contractive truncated unrolled solver and backpropagating only through the final block yields gradients arbitrarily close to the deep-equilibrium hypergradient.","lead":"Researchers introduce ReTune, a training procedure for unrolled optimization networks that repeatedly applies a short truncated solver and updates hyperparameters by backpropagating only through the final block. They prove this cheap gradient can get arbitrarily close to the full deep-equilibrium gradient when each block is a contraction, and they show it can tune denoisers and regularization parameters in imaging tasks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Contraction constant δK(θ) is never measured; the bound in Theorem 2 may be vacuous at the reported K=T=10, leaving the claimed closeness to DEQ unverified in the one experiment designed to satisfy the theory.","rationale":"The reader's weakest_assumption is exactly Assumption 1, and the reader notes that δK is not measured numerically. I agree this is the most load-bearing condition: every theorem in Section 3 scales with δK or δK^T, so an unverified contraction constant that is merely asserted to be below 1 leaves the quantitative central claim untested. The additional indexing slip in the proof of Theorem 2 is real but secondary; it makes the stated bound optimistic by 1/δK yet does not affect the qualitative 'arbitrarily close' conclusion. The paper is honest about where assumptions fail, and the method is clearly described, so conditional acceptance remains appropriate; the revision should add a numerical estimate of δK(θ) in the wavelet experiment (and ideally in the PnP experiments where the theory is admittedly not satisfied). This would either strengthen the claimed theoretical support or reveal the experiments to be heuristic, which the conditional verdict already tolerates.","tokens_in":17638,"tokens_out":20322,"duration_ms":215235,"concrete_test":"On the trained wavelet model, estimate δK(θ*) at the final learned parameters θ* by power iteration on Jacobian-vector products of ΦK(·, θ*) (or by random finite differences), then evaluate the dominant terms of Corollary 1's bound with K=T=10. If the computed RHS is not small compared with the actual distance ∥g(θ*)−gR(θ*)∥ (e.g., estimated by DEQ on a few samples), the theoretical bound is not active in the matching experiment and the claim that numerics support Theorem 2 fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central result, Theorem 2, bounds the ReTune/DEQ gradient error by the JFB term δK/(1−δK)∥∂xL∥∥∂θΦK∥ plus restart terms that scale as δK^T. These bounds are only useful—and the 'arbitrarily close' conclusion only non-vacuous—if δK(θ) is substantially below 1. Assumption 1 merely postulates δK(θ)<1; the wavelet experiment intended to match the theory asserts strong convexity and a stepsize choice, but reports no numerical estimate of δK(θ). Because the learned parameters θ are unconstrained (only positivity is enforced), the contraction ratio can approach 1 (e.g., when min(θ) is small), making δK^10 and δK/(1−δK) large. At δ=0.99, δK^10≈0.90 and the JFB factor ≈9.4, so the bound cannot explain the observed PSNR gains at K=T=10. The theoretical support claimed for the numerics is therefore not established. (Secondary: the proof of Theorem 2 in lines (40)–(41) bounds ∥bxθ−xK(T−1)∥ by δK^T; Theorem 1 gives δK^{T−1}, so the stated bound (33) is optimistic by a factor 1/δK.)","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper considers bilevel optimization for imaging inverse problems, where the inner problem is solved by an iterative scheme truncated to K steps, called a truncated unrolled scheme. The authors propose ReTune: apply T restarts of the K-step operator and compute the hypergradient by automatic differentiation through only the last restarted block, avoiding the Jacobian inversion required by the Deep Equilibrium (DEQ) framework. The main theoretical results are: (i) Theorem 1, convergence of the restarted iterates to the inner solution under a contraction assumption on the K-step operator; (ii) Lemma 1, a bound on the error between the exact DEQ gradient and the Jacobian-Free Backpropagation (JFB) gradient at the fixed point; (iii) Theorem 2, a bound on the error between the DEQ gradient and the ReTune gradient, showing the error tends to zero as the number of restarts increases, up to a JFB-type term that decays with K. Numerical experiments on wavelet denoising, inpainting, and deblurring withPlug-and-Play denoisers show that ReTune with K=T=10 yields improved training PSNR relative to unrolled and non-restarted baselines.","tokens_in":17979,"tokens_out":6297,"duration_ms":75829,"significance":"The paper proposes a clean, practically motivated link between truncated unrolled networks and DEQ/JFB implicit differentiation. The derivation is transparent and does not fit any constants to data; the assumptions are stated explicitly, and the dependence of the error bounds on the contraction constant δK and the restart count T is made precise. If the proof issues are corrected, the asymptotic guarantee that ReTune gradient steps approximate DEQ steps is a useful theoretical justification for a simple and memory-efficient training scheme. The empirical section honestly acknowledges that several experiments depart from the theoretical assumptions, and the results suggest practical robustness beyond the verified regime. The main weakness is that the one experiment intended to satisfy the theory does not measure the contraction constant, so the quantitative connection between theory and numerics remains unverified.","major_comments":[{"comment":"The stated bound in Eq. (33) is optimistic by a factor 1/δK in the second term. In the proof, the term ∥∂xL(bxθ)∥2 Lθ∥bxθ − xK(T−1)∥2 is bounded by δK^T∥∂xL(bxθ)∥2 Lθ∥bxθ − x0∥2, but Theorem 1 gives ∥bxθ − xK(T−1)∥2 ≤ δK^{T−1}∥bxθ − x0∥2, not δK^T. Moreover, the argument of ∂θΦK in (36) is xK(T−1), so Assumption 2 must be applied at xK(T−1), not at xKT as suggested by the bound in (40). The corrected term should carry δK^{T−1}. The qualitative conclusion that all restart terms vanish as T grows is unaffected, but the statement and proof of Theorem 2 must be corrected to be mathematically accurate.","section":"Section 3.3, Eq. (33) and proof lines (40)-(41)"},{"comment":"The wavelet denoising experiment is the only one claimed to satisfy Assumption 1, yet the contraction constant δK(θ) is never measured or reported. Since the learned parameters are only constrained to be positive (via exp), the ratio min(θ)/max(θ) can become small, making δK(θ) approach 1 despite the fixed stepsize τ = 1.95/L. At K=T=10, if δK were close to 1, the bounds in Theorem 2 would be vacuous and could not explain the observed PSNR gains. The paper should either report an empirical estimate of δK(θ) during training (or at least a uniform upper bound derived from the parameter ranges) or explicitly state that the experiment does not verify the quantitative tightness of the bound.","section":"Section 4.1, 'Validity of the assumptions' and Figure 1"},{"comment":"The inpainting and deblurring experiments use non-injective forward operators and a DRUNet denoiser, which the paper acknowledges do not satisfy Assumption 1. The conclusion nevertheless states that the theoretical analysis is 'supported by numerical experiments.' This is acceptable as a claim of robustness, but the wording should be sharpened: experiments outside the assumptions cannot provide evidence for the specific quantitative bounds of Theorem 2; they only suggest that the ReTune strategy remains effective in regimes not covered by the theory. I recommend rephrasing the conclusion to distinguish 'illustrating the method outside the theoretical scope' from 'validating the theory.'","section":"Section 4.2 and Conclusion"}],"minor_comments":[{"comment":"The displayed bound in Eq. (33) is typeset incorrectly: the last two lines contain a dangling brace and a missing multiplication symbol. The expression should be cleaned up to make the o(·) term and the δK^T factor unambiguous.","section":"Section 3.3, Eq. (33)"},{"comment":"The theorem statement does not mention that T must be large enough for xK(T−1) to lie in the neighborhood where Assumption 2 holds. The proof notes this, but the statement should include the quantifier, e.g., 'for all T sufficiently large'.","section":"Section 3.3, Theorem 2"},{"comment":"The sentence 'φk is (δ <1)-Lipschitz continuous' should give the explicit formula δ ≤ ω = max{|1−τ μ|, |1−τ L|} with the chosen stepsize, and state whether the bound is uniform over the parameter space explored during training. Without this, the reader cannot assess how close δK is to 1.","section":"Section 4.1, 'Validity of the assumptions'"},{"comment":"The legends in Figures 2 and 3 label configurations as (K,T) = (1,1), (1,10), (10,1), (10,10), but Figure 3 uses 'T: 0, K: 10' and 'T: 0, K: 1' in the inset. This is inconsistent with the text and should be corrected to T=1.","section":"Figures 2 and 3"}],"recommendation":"major_revision","confidential_remarks":"The central theoretical idea is sound and the paper is within the scope of math.OC. The off-by-one error in Theorem 2 is fixable and does not change the asymptotic message, but it must be corrected because the theorem is the paper's main contribution. The numerical verification gap regarding δK is the other substantive issue; I would like to see at least one quantitative estimate of the contraction ratio in the wavelet experiment, or a clear statement that the experiments are not intended to verify the bound. Overall, the paper is promising and I recommend acceptance after these revisions."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a solid, useful paper. ReTune is a small but genuine algorithmic idea — truncate, restart, and backprop only through the last block — and the paper proves something the literature has only assumed: the restarted-truncated gradient is close to the DEQ gradient, with an explicit bound separating JFB error (δ/(1−δ)) from restart error (δ^T). That is worth having. The proof is a clean assembly of Neumann bounds and Lipschitz estimates, and the statements are precise. The wavelet experiment is designed to satisfy the theory, and the authors are upfront that the PnP and deblurring experiments depart from the assumptions. Credit where due.\n\nSoft spots, in order of importance. First, the proof of Theorem 2 has an indexing slip: the Lipschitz term for ∂θΦK is evaluated at x_{K(T−1)}, so Theorem 1 gives δ^{T−1}‖x0−bxθ‖, not δ^T. The stated bound is optimistic by a factor 1/δ, but the qualitative conclusion still holds. Second, the contraction constant δK(θ) is never measured, even in the wavelet experiment meant to validate the theory. Since learned weights are only constrained to be positive, δ can easily sit near 1; at δ=0.99 and K=T=10 the bound is too loose to explain the observed PSNR gains. The theory motivates the method, but the paper does not verify the quantitative regime where the bound actually bites. Third, the numerical section compares ReTune configurations against each other only — no DEQ or other bilevel baseline, no error bars, no code. That weakens the empirical claims but does not damage the theoretical contribution.\n\nWho this is for: people working on learned optimization, deep equilibrium training, and imaging inverse problems. It deserves a serious referee. I would send it to review and expect a revision that fixes the indexing slip, measures or bounds δK in the matching experiment, and adds at least one DEQ/JFB comparison. With that, I would accept it.","headline":"A genuinely useful quantitative bridge between truncated backprop and DEQ gradients, let down by an unmeasured contraction constant and thin baselines.","tokens_in":18514,"tokens_out":2759,"would_cite":true,"duration_ms":31527,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["65K10","90C25","49J52","68T07"],"pacs":[],"model":"deepseek-v4-flash","headline":"Backpropagating through only the last block of a restarted contractive solver can be made arbitrarily close to the full deep-equilibrium gradient step.","keywords":["bilevel optimization","deep equilibrium models","Jacobian-free backpropagation","unrolled neural networks","contractive fixed-point operators","plug-and-play imaging","hyperparameter learning","restarted iterative schemes"],"falsifier":"Measure the empirical contraction factor $\\delta_K$ or the operator norm of $\\partial_x \\Phi_K(\\cdot,\\theta)$ on the wavelet-denoising training trajectory; if any visited $\\theta$ has $\\delta_K \\ge 1$, or if on a case with known $\\delta_K<1$ the gap between $g_R$ and the exact DEQ gradient $g$ fails to shrink like $\\delta_K^T$ as T increases, the theorem's bound (33) is contradicted.","tokens_in":17406,"feed_emoji":"🔁","tokens_out":8969,"duration_ms":81941,"temperature":0.7,"pith_summary":"The paper proves that a cheap surrogate gradient — Jacobian-free backpropagation through the final block of a restarted truncated solver — can be made arbitrarily close to the exact deep-equilibrium hypergradient. Theorem 2 bounds the difference by a term that shrinks with the K-step contraction rate plus terms that decay exponentially in the number of restarts T. This matters for imaging inverse problems because it justifies training unrolled networks without forming or inverting a high-dimensional Jacobian. The authors validate the approach on wavelet denoising inside the theoretical assumptions, then on inpainting and deblurring with pretrained and learned denoisers, where restarts still help even though the assumptions are relaxed.","feed_headline":"One restarted block can replace full implicit-differentiation gradient","feed_subtitle":"ReTune provably drives truncated unrolled learning's gradient step as close as desired to the DEQ step, with no Jacobian inversion.","key_machinery":"The load-bearing object is the restarted truncated operator $\\Phi_K^T(x_0,\\theta) = \\Phi_K(\\cdot,\\theta) \\circ \\cdots \\circ \\Phi_K(\\cdot,\\theta)$ (T times), where $\\Phi_K$ is K steps of a contractive proximal algorithm such as forward-backward splitting. ReTune runs these restarts and applies automatic differentiation only to the final step, producing $g_R$. The proof mechanism is the Neumann-series bound $\\|I-(I-H)^{-1}\\|_2 \\le \\omega/(1-\\omega)$ for $\\|H\\|_2 = \\omega < 1$, which converts the contraction rate into the Jacobian-free error, together with linear convergence of the restart iterates to the fixed point and a local Lipschitz condition on $\\partial_\\theta \\Phi_K$ controlling the remaining terms.","core_discovery":"The central claim is that for a parameterized fixed-point scheme whose K-step composition is a contraction, the DEQ gradient step $g(\\theta) = \\partial_x L(\\bar{x}_\\theta)^\\top [I - \\partial_x \\Phi_K(\\bar{x}_\\theta, \\cdot)]^{-1} \\partial_\\theta \\Phi_K(\\bar{x}_\\theta, \\cdot)(\\theta)$ can be replaced by the last-block gradient $g_R(\\theta) = \\partial_x L(x_{KT})^\\top \\partial_\\theta \\Phi_K(x_{K(T-1)}, \\cdot)(\\theta)$ with a quantified error. The error decomposes into a Jacobian-free-backpropagation error bounded by $\\delta_K/(1-\\delta_K)$ and restart errors that decay like $\\delta_K^T$, where $\\delta_K<1$ is the Lipschitz constant of the K-step operator. Hence by increasing the unrolled depth K and the number of restarts T, the learned update is provably as good as the full implicit-differentiation step, without ever computing $[I - \\partial_x \\Phi_K]^{-1}$. This is the sense in which ReTune learns at equilibrium.","pith_inferences":["Going beyond the paper, the same restart-plus-JFB argument should apply to other contractive fixed-point solvers, such as Douglas-Rachford or primal-dual schemes, whenever the composed operator's parameter derivative is locally Lipschitz.","The theory suggests an adaptive budget in which K and T grow during training, or are set from an estimated $\\delta_K$, trading memory for gradient accuracy; the experiments only test fixed choices such as $K,T \\in \\{1,10\\}$.","The success on non-contractive Plug-and-Play denoisers hints that a local or statistical contraction near the fixed point may be enough, and a testable extension would measure the empirical Lipschitz constant along the training trajectory."],"forward_implications":["Larger K shrinks the Jacobian-free component of the gradient error to zero, so depth can serve as a substitute for inverting the DEQ Jacobian.","Larger T drives the restart point $x_{KT}$ to the fixed point $\\bar{x}_\\theta$, so the inner problem is solved exactly while backpropagation stays limited to one K-step block.","The bound (33) gives a principled resource split: increase K to reduce the JFB error and increase T to reduce the restart error.","For strongly convex forward-backward problems satisfying the assumptions, ReTune is a provably convergent bilevel learning procedure, unlike standard truncated unrolled training."],"supporting_citations":[{"why":"Introduces Jacobian-free backpropagation, the surrogate gradient whose error Lemma 1 quantifies and which ReTune evaluates at a restart point.","marker":"[15]"},{"why":"Defines the deep equilibrium fixed-point formalism whose gradient (20) is the reference step the paper proves ReTune approximates.","marker":"[2]"},{"why":"Brings deep equilibrium architectures to imaging inverse problems, the application domain of the numerical experiments.","marker":"[16]"},{"why":"Supplies the Banach-Picard theorem and forward-backward convergence analysis used in Theorem 1 and in Assumption 1.","marker":"[3]"},{"why":"Provides the Neumann-series estimate $\\|I-(I-H)^{-1}\\|_2 \\le \\omega/(1-\\omega)$ that produces the JFB error bound in Lemma 1 and Theorem 2.","marker":"[32]"},{"why":"Gives the exact contraction rate of strongly convex proximal-gradient steps, grounding the condition $\\delta_K(\\theta)<1$.","marker":"[38]"},{"why":"Supplies the pretrained neural denoiser used in the Plug-and-Play experiments that test ReTune outside the theoretical assumptions.","marker":"[39]"}],"fun_headline_variants":["Restarted contraction replaces Jacobian inversion in equilibrium learning","Learn at equilibrium: restarts make implicit gradient needless","ReTune: restarted contractive operators learn without inverting Jacobians","No Jacobian inverse needed: restarts achieve equilibrium learning","Restarting contraction yields invert-free equilibrium learning"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is Assumption 1: for every parameter value, the K-step operator must be a strict contraction with Lipschitz constant below one and a unique fixed point, and if that fails the convergence theorem and every error bound collapse.","fun_headline_variants_meta":{"raw":{"variants":["Restarted contraction replaces Jacobian inversion in equilibrium learning","Learn at equilibrium: restarts make implicit gradient needless","ReTune: restarted contractive operators learn without inverting Jacobians","No Jacobian inverse needed: restarts achieve equilibrium learning","Restarting contraction yields invert-free equilibrium learning"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000357,"raw_usage":{"total_tokens":2003,"prompt_tokens":1079,"completion_tokens":924,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":695,"completion_tokens_details":{"reasoning_tokens":842}},"tokens_in":695,"tokens_out":924,"duration_ms":9701,"temperature":1.0,"reasoning_tokens":842,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T00:37:48.376264+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Measure the empirical contraction factor $\\delta_K$ or the operator norm of $\\partial_x \\Phi_K(\\cdot,\\theta)$ on the wavelet-denoising training trajectory; if any visited $\\theta$ has $\\delta_K \\ge 1$, or if on a case with known $\\delta_K<1$ the gap between $g_R$ and the exact DEQ gradient $g$ fails to shrink like $\\delta_K^T$ as T increases, the theorem's bound (33) is contradicted.","supporting_citations":[{"cited_title":"JFB: Jacobian-free backpropagation for implicit networks","cited_arxiv_id":null,"evidence_quote":"Introduces Jacobian-free backpropagation, the surrogate gradient whose error Lemma 1 quantifies and which ReTune evaluates at a restart point."},{"cited_title":"Zico Kolter, and Vladlen Koltun","cited_arxiv_id":null,"evidence_quote":"Defines the deep equilibrium fixed-point formalism whose gradient (20) is the reference step the paper proves ReTune approximates."},{"cited_title":"Deep equilibrium architectures for inverse problems in imaging","cited_arxiv_id":null,"evidence_quote":"Brings deep equilibrium architectures to imaging inverse problems, the application domain of the numerical experiments."},{"cited_title":"Bauschke and Patrick L","cited_arxiv_id":null,"evidence_quote":"Supplies the Banach-Picard theorem and forward-backward convergence analysis used in Theorem 1 and in Assumption 1."},{"cited_title":"Translated by Leo F","cited_arxiv_id":null,"evidence_quote":"Provides the Neumann-series estimate $\\|I-(I-H)^{-1}\\|_2 \\le \\omega/(1-\\omega)$ that produces the JFB error bound in Lemma 1 and Theorem 2."},{"cited_title":"Taylor, Julien M","cited_arxiv_id":null,"evidence_quote":"Gives the exact contraction rate of strongly convex proximal-gradient steps, grounding the condition $\\delta_K(\\theta)<1$."},{"cited_title":"Plug- and-play image restoration with deep denoiser prior","cited_arxiv_id":null,"evidence_quote":"Supplies the pretrained neural denoiser used in the Plug-and-Play experiments that test ReTune outside the theoretical assumptions."}],"review_version":1}