{"id":"1054e096-cc95-453a-a2d7-6a752651bdac","arxiv_id":"2506.14263","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"The authors prove that out-of-distribution convergence degradation in learning to optimize scales with the magnitude of the model's input feature shift, and build a gradient-only optimizer that reduces this shift.","lead":"This paper gives convergence bounds for learned optimizers under distribution shift and introduces a gradient-only learned optimizer, GO-Math-L2O, that shrinks the input features those bounds depend on. A generalist might read it because it tries to turn learning to optimize from an empirical trick into a method with robustness guarantees, relevant wherever deployed solvers meet out-of-distribution problems.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The OOD-deterioration theorems rest entirely on Assumption 1, which requires a trained L2O to equal 1/L gradient descent at every InD point; this is neither verified nor satisfied by GO-Math-L2O's own update, so the derived bounds need not apply to the evaluated models.","rationale":"The reader's weakest assumption is Assumption 1, and I agree that the exact post-training equality to 1/L gradient descent is the load-bearing point. I sharpen this in two ways. First, the assumption is not merely unverified; the paper's own GO-Math-L2O architecture contradicts the assumed update form, since its smooth-limit update has no N2 bias and its R_k is a learned bounded matrix. The corollaries that motivate the architecture are therefore formally about a different model. Second, even granting Assumption 1, the results are upper bounds, not lower bounds; they show that OOD decrease is no better than a possibly positive expression, but they do not prove that OOD convergence deteriorates or that reducing ||s'|| improves actual performance. The abstract's wording, and the claim that Corollaries 2 and 3 demonstrate outperformance over [14], overstate what the inequalities establish. A conditional verdict remains appropriate: the framework is a plausible sufficient-condition analysis, and the proposed architectural direction is reasonable, but the central claim is not established for the trained models until the gap between Assumption 1/3 and real networks is quantified or the theory is reworked with perturbation terms. The concrete test would settle whether the gap is small enough to be negligible, or whether the idealization is doing the work.","tokens_in":72133,"tokens_out":9276,"duration_ms":104289,"concrete_test":"Train L2O-PA and GO-Math-L2O exactly as in Sec. 6, then freeze the networks and, on 1,000 held-out InD LASSO instances, sample iterates k=1,...,100. Compute delta(x)=||d(z) - grad f(x)/L|| for each model and the deviations of N1,N2 from Assumption 1; report max and mean relative to ||grad f(x)/L||. If delta exceeds 10% of the GD step norm, Assumption 1 is empirically false and Theorems 1-2 and Corollaries 2-3 are not applicable to those trained models. As a second arm, recompute the Corollary 2 RHS using measured N1,N2 with a first-order error term and check whether the claimed ordering of L2O-PA versus GO-Math-L2O still holds; if not, the theoretical tightening argument is unsupported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Assumption 1 (Sec. 4.1) fixes N1(z)=1/(2L)I and N2(z)=grad f(x)/(2L) pointwise for all InD z, so d(z)=grad f/L. Every subsequent claim, Theorem 1, Corollary 2, Theorem 2, and Corollary 3, starts from this exact equality. A neural network trained with Adam on 32,000 finite problems cannot be certified to satisfy a continuum of pointwise equalities, and the paper reports no check of update-versus-GD error. This is not a minor constant: if N1 or N2 deviates, the cancellations of the linear J terms in eqs. (24) and (26) fail, and the deterioration bound gains model-dependent error terms that need not be controlled by ||s'||.\n\nThe issue is sharper for GO-Math-L2O. In the smooth limit of Theorem 3, the update is x_k = x_{k-1} - R_k grad f(x_{k-1}) with b1=0, v=0, g=0: there is no N2 bias term at all. Assumption 1's N2 = grad f/(2L) cannot hold for this architecture, and R_k is a learned Sigmoid-bounded matrix rather than the constant 1/(2L)I. Thus Corollaries 2 and 3, which are the stated theoretical reason that gradient-only features tighten the bound, are not theorems about the model evaluated in Sec. 6. Section 11's Assumption 3 is a different idealized model (N1=1/L, N2=N3=N4=0) and is also unverified.\n\nAdditionally, the step from these upper bounds to deterioration is an overreach: Theorem 1 bounds F'(x_k+s_k) - F'(x_{k-1}+s_{k-1}) from above by -||grad f'||^2/(2L) plus positive terms. An upper bound with positive terms does not imply the actual decrease is smaller than GD's; it only says the decrease is no better than a possibly positive value. Corollary 3's RHS is not a O(1/K) rate, since the leading distance terms are not divided by K and the s' terms scale with trajectory norms. Reducing ||s'|| tightens an upper bound, but it does not by itself establish faster convergence or outperformance over [14].","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper aims to provide theoretical guarantees for learning-to-optimize (L2O) models in out-of-distribution (OOD) scenarios. It introduces a 'virtual feature' and 'virtual trajectory' to relate an OOD optimization trajectory to an in-distribution (InD) trajectory, and proves several bounds on the per-iteration and multi-iteration convergence of the L2O model in the OOD setting under an assumption that the learned optimizer exactly matches gradient descent on all InD instances. Based on these bounds, the paper claims that larger input feature magnitudes cause greater deterioration of convergence, and proposes a new gradient-only L2O model, GO-Math-L2O, with a history modeling mechanism. Experiments on LASSO and logistic regression, including synthetic and real-world OOD scenarios, show that GO-Math-L2O often converges faster than existing L2O baselines.","tokens_in":72582,"tokens_out":11447,"duration_ms":105663,"significance":"The empirical contribution is solid and the code is provided. The idea of linking OOD behavior to input feature magnitude is interesting and could inspire future work. However, the theoretical foundations are not established: the key assumption is an unverified exact equality, the derived upper bounds do not logically imply deterioration, and the rate statements contain algebraic errors. As a result, the paper's main theoretical claim is not supported, and the theory does not apply to the proposed model.","major_comments":[{"comment":"Assumption 1 (Sec. 4.1) postulates N1(z_{k-1}) = 1/(2L) I and N2(z_{k-1}) = ∇f(x_{k-1})/(2L) pointwise on all InD inputs. The paper does not show that a network trained with Adam on 32,000 problems satisfies this continuum of equalities, nor does it report any check of the deviation. More importantly, the proposed GO-Math-L2O model (Eqs. (8) and (10)) has no N2 bias term; even in the smooth limit with v=0 and b1=0, its update is x_k = x_{k-1} - R_k ∇f(x_{k-1}) with a learned matrix R_k, so Assumption 1 cannot hold for this architecture. Consequently, Theorems 1, 2 and Corollaries 2, 3, which are presented as the theoretical motivation for gradient-only features, do not apply to the model evaluated in Section 6.","section":"Sec. 4.1, Assumption 1; Sec. 5"},{"comment":"Theorem 1 bounds F'(x_k+s_k) - F'(x_{k-1}+s_{k-1}) above by -||∇f'||^2/(2L) plus nonnegative residual terms. An upper bound with positive terms does not imply the actual decrease is smaller than gradient descent's; the actual decrease could be larger (more negative). Corollary 2 and 3 similarly bound the objective difference above by quantities that increase with ||s'||, but these are not lower bounds. The paper's statements that 'any OOD will dampen convergence' and that 'Corollaries 2 and 3 theoretically demonstrate the outperformance over the method in [14]' are therefore not logical consequences of the results. A lower bound or a worst-case construction is needed to substantiate the deterioration claim.","section":"Sec. 4.1, Theorem 1 and Corollaries 2-3"},{"comment":"In Theorem 2 and Corollary 3, the first two terms on the right-hand side, L/2 ||x0 - x* + s0 - s*||^2 - L/2 ||xK - x* + sK - s*||^2, are not multiplied by 1/K, while the sum terms are. Since these bounds are derived by summing K per-iteration inequalities and then dividing by K to pass to the minimum, the leading terms must also be divided by K. As stated, the right-hand side is O(1) in K and cannot serve as a convergence rate. This is a load-bearing algebraic error in the paper's main theoretical results.","section":"Sec. 4.1, Theorem 2 and Corollary 3"},{"comment":"Theorem 3 claims existence of symmetric positive definite R_k, Q_k, B_k under Conditions FP and GC. In the proof, R_k is defined as the Jacobian block J_{2,k} of a mean-value representation; its positive definiteness is not established from FP and GC. The argument later 'assumes' R_k ≻ 0 to derive the proximal operator form (Eq. (11)). The theorem as stated is therefore not proven; at best it proves existence of bounded matrices with an additional unstated regularity condition.","section":"Sec. 5, Theorem 3"}],"minor_comments":[{"comment":"There are numerous typos and grammatical errors in the appendix, including 'inversible', 'contruct', 'Firgure', 'derease', 'entra', and 'objeective'. A careful proofreading pass is needed.","section":"Throughout"},{"comment":"The text says the gradient-only feature reduces the input magnitude 'by 1/3', but the change from [x^T, ∇f^T]^T (2n dimensions) to ∇f (n dimensions) is a reduction by 1/2, not 1/3.","section":"Sec. 5"},{"comment":"The sentence 'Note that s := 0 cannot lead to the convergence rate of gradient descent since the third term in equation 34 is non-zero and cannot be eliminated' appears to contradict the immediately preceding derivation, which shows cancellation for s=0. This passage should be reworded for clarity.","section":"Sec. 8.6 (Appendix)"},{"comment":"The statement that 'Corollaries 2 and 3 theoretically demonstrate the outperformance over the method in [14]' is an overreach given the upper-bound nature of the results; a larger upper bound does not constitute a proof of worse performance.","section":"Sec. 2.2"},{"comment":"The legends have duplicate entries (e.g., 'GO-Math-L2O, s = 10' appears twice), presumably one curve for +s and one for -s. The captions should clarify which line corresponds to which sign.","section":"Figures 3, 4, 16, 17"},{"comment":"The real-world OOD experiments in Figures 2, 14, and 15 are not clearly defined as OOD; the paper should specify the distribution shift relative to the training distribution.","section":"Sec. 6.2"}],"recommendation":"reject","confidential_remarks":"The paper has a substantial empirical component (GO-Math-L2O with strong experimental results) that could be published separately as a systems/empirical contribution. However, the theoretical analysis as presented is not sound: the central deterioration claim is not supported by the upper-bound results, the proposed model does not satisfy the key assumption, and the rate statements contain algebraic errors. I recommend reject, though a major revision that drops or substantially reworks the theoretical claims and focuses on the empirical findings might be reconsidered."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the virtual-feature machinery is new and worth knowing; the deterioration theorems are internally consistent but rest entirely on Assumption 1, which is not verified and is not satisfied by the paper's own GO-Math-L2O update. The paper also reads its upper bounds as proof of outperformance, which does not follow. Treat the theory as conditional and the empirical section as the stronger half.\n\nWhat is genuinely new: defining virtual feature s' = z' - z and a virtual Jacobian to relate OOD and InD trajectories is a useful way to quantify OOD shift for L2O. The gradient-only history model, recovering subgradients from an invertible update, is a reasonable design consequence and a real architectural contribution relative to Math-L2O. The experiments are carefully reported: code link, training configurations, ablations, and real-world datasets. I believe the empirical speedups are real.\n\nSoft spots. First, Assumption 1 fixes N1 = 1/(2L) I and N2 = grad f / (2L) pointwise for every InD input. That is exactly 'the trained model is 1/L gradient descent everywhere in distribution.' No check is reported that a trained network satisfies this. More importantly, GO-Math-L2O's smooth-limit update is x_k = x_{k-1} - R_k grad f with a learned Sigmoid-bounded R_k and no N2 term, so Assumption 1 cannot hold for it. Therefore Corollaries 2 and 3, the stated theoretical reason gradient-only features help, are not theorems about the model evaluated in Section 6. The longer-horizon Assumption 3 is a different idealization and is also unverified.\n\nSecond, the move from upper bounds to outperformance is overreach. Theorem 1 gives F'(x_k+s_k) - F'(x_{k-1}+s_{k-1}) <= -||grad f'||^2/(2L) plus positive terms. An upper bound with positive terms does not imply the actual decrease is worse than gradient descent's; it only bounds it above by a possibly positive number. Corollary 3's RHS is not a O(1/K) rate; the leading distance terms are not divided by K, and the s' terms scale with trajectory norms. Reducing ||s'|| tightens a bound; it does not establish faster convergence.\n\nNone of this kills the paper. The virtual-feature idea and the empirical model can survive; the theory section needs either verification or relaxation of Assumption 1, or an honest reframing as an idealized-condition analysis. A serious referee should engage. I would send it out, with a strong request to fix the claim mismatch between the theory and the evaluated model.","headline":"A genuinely new virtual-feature OOD analysis for L2O that is internally consistent under a very strong 'trained model equals gradient descent' assumption, but the proposed GO-Math-L2O does not satisfy that assumption, and the upper bounds are over-read as superiority proofs.","tokens_in":73224,"tokens_out":2615,"would_cite":false,"duration_ms":26127,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["68T07","90C25"],"pacs":[],"model":"deepseek-v4-flash","headline":"A learning-to-optimize model that is perfectly trained as gradient descent on every in-distribution instance still has no proved convergence guarantee on out-of-distribution instances; the paper bounds the deterioration by the magnitude…","keywords":["learning to optimize","out-of-distribution generalization","convergence rate","virtual feature","gradient-only features","GO-Math-L2O","convex optimization","robustness"],"falsifier":"Train a one-dimensional convex quadratic problem with an L2O model constructed so that Assumption 1 holds on a dense InD grid, then run the frozen model on an OOD objective $F'(x)=F(x+t)$ and check whether the per-iteration objective drop ever exceeds the right-hand side of Corollary 2; any violation would refute the bound. Alternatively, evaluate $N_1(z)$ and $N_2(z)$ on held-out InD points and find any point where they differ from $1/(2L)$ and $\\nabla f/(2L)$, which would show that the theorem's premise is not satisfied in practice.","tokens_in":71798,"feed_emoji":"📉","tokens_out":7261,"duration_ms":71414,"temperature":0.7,"pith_summary":"This paper tries to make robustness to out-of-distribution (OOD) shifts a provable property of learned optimizers instead of an empirical hope. It first identifies a sufficient condition under which a learning-to-optimize (L2O) model improves in every iteration on all in-distribution (InD) instances: the learned update must coincide with gradient descent with step $1/L$. Assuming that condition holds exactly, the paper derives upper bounds showing that OOD per-iteration progress deteriorates from the gradient-descent improvement $-\\|\\nabla f'\\|^2/(2L)$ by positive terms proportional to the magnitude of a 'virtual feature,' the difference between the model's input features in OOD and InD scenarios; the same structure appears for $K$-iteration convergence rates. Because the bounds grow with feature size, the paper proposes an L2O model with gradient-only features, GO-Math-L2O, and reports that it outperforms state-of-the-art baselines in both InD and OOD tests, with up to $10\\times$ faster convergence in OOD scenarios. A sympathetic reader should take away a precise statement: input feature construction is not a modeling detail but a first-order determinant of OOD convergence guarantees.","feed_headline":"Perfect L2O training doesn't guarantee OOD convergence","feed_subtitle":"A proved bound ties out-of-sample slowdown to the size of the optimizer's input features.","key_machinery":"The load-bearing object is the virtual feature $s' = z' - z$, the difference between L2O input features in OOD and InD scenarios at aligned points, together with the mean-value expansion $d(z')=d(z)+J_d s'$ using a bounded virtual Jacobian $J_d$. This expansion connects each OOD trajectory to an InD trajectory whose convergence behavior is known, and then bounds the gap. The quantitative engine is the per-iteration bound of Theorem 1 and its $\\|s'\\|$-dependent corollaries, combined with the sufficient condition (Assumption 1) that the trained network equals gradient descent, $N_1=1/(2L)$ and $N_2=\\nabla f/(2L)$, on all InD points.","core_discovery":"The paper's central claim is that an L2O model's OOD convergence rate deteriorates by an explicit function of its input features. Under Assumption 1, where after training $N_1(z)=1/(2L)$ and $N_2(z)=\\nabla f(x)/(2L)$ on every InD point, Theorem 1 bounds one iteration's OOD improvement by $-\\|\\nabla f'(x+s)\\|^2/(2L)+L\\|\\mathrm{diag}(J_1s')\\nabla f'(x+s)\\|^2+L\\|(\\nabla f'(x+s)-\\nabla f(x))/(2L)-J_2s'\\|^2$, and Corollary 2 relaxes this to explicit positive terms in $\\|s'\\|^2$. Theorem 2 and Corollary 3 extend the pattern to $K$ iterations: the gradient-descent telescoping terms remain, but additional sums proportional to $\\|s'_{k-1}\\|$ prevent any guaranteed global convergence. The paper also argues the same deterioration mechanism holds for non-smooth and longer-horizon settings, and that eliminating variable-related features shrinks $s'$ and therefore tightens every bound. Numerically, the resulting gradient-only GO-Math-L2O converges faster than prior learned and classical baselines in the paper's test problems.","pith_inferences":["The paper's Assumption 1 is a pointwise equality, and real networks trained by Adam on finite datasets almost surely violate it; the tightest reading of the theorems applies to an idealized perfectly trained optimizer rather than to every trained checkpoint.","If the input-feature dependence is causal as stated, then L2O robustness benchmarks could report measured feature drift $\\|s'\\|$ alongside objective trajectories; one could test whether models with similar in-distribution accuracy but smaller feature drift exhibit the predicted smaller OOD gap.","The same virtual-feature pairing may extend beyond L2O to any learned fixed-point iterator: the bound formalizes a generic price for distribution shift in the input representation, with the Jacobian norm of the learned map as the key constant."],"forward_implications":["An L2O model that behaves exactly as gradient descent on all in-distribution instances still admits no proved convergence guarantee on out-of-distribution instances; the derived upper bounds contain positive terms that can outweigh the negative gradient-descent improvement.","The deterioration in the OOD convergence bound scales with the magnitude of the L2O model's input features $\\|s'\\|$, and this is asserted as a theorem-level statement rather than an empirical trend.","Feature designs that eliminate variable-related inputs reduce the dimension and magnitude of $s'$ and thereby tighten the OOD bound for both one iteration and $K$ iterations.","For composite objectives, a gradient-only longer-horizon construction preserves the same theoretical structure, and the paper reports that GO-Math-L2O converges up to $10\\times$ faster than state-of-the-art baselines in OOD numerical tests.","Because the InD model is assumed to be exactly gradient descent, in-distribution convergence can be guaranteed at the rate $L/(2K)\\|x_0-x^*\\|^2$, while the OOD bound retains extra positive terms that are not guaranteed to vanish."],"supporting_citations":[{"why":"Defines the Math-L2O update form and workflow that the paper builds on, and supplies the baseline L2O-PA used in experiments.","marker":"[14]"},{"why":"Supplies gradient descent, the baseline algorithm whose $1/L$-step convergence rate appears in Corollary 1 and in the negative term of the OOD bounds.","marker":"[19]"},{"why":"Defines FISTA, used both to generate optimal objective labels for the experimental evaluation and as a classical comparison baseline.","marker":"[5]"},{"why":"Adam is the optimizer used to train the proposed model and the learning-based baselines.","marker":"[13]"},{"why":"Ada-LISTA is one of the learned-optimizer baselines in the experimental comparison.","marker":"[1]"},{"why":"L2O-DM is a black-box learning-to-optimize baseline included in the experimental comparison.","marker":"[3]"},{"why":"L2O-RNNprop is a baseline learned optimizer compared in the experiments.","marker":"[15]"},{"why":"The proximal point algorithm is the backbone used in the non-smooth and composite convergence analyses.","marker":"[18]"}],"fun_headline_variants":["OOD convergence bound depends on L2O feature size","Gradient-only L2O: theory-backed 10x speedup","Proved: OOD slowdown scales with input features in L2O","Robust L2O with guarantees: smaller features, faster OOD","New L2O method: tighter bounds, 10x faster convergence"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper's OOD deterioration equations suppose that after training the learned optimizer is exactly gradient descent with step $1/L$ at every in-distribution point; if a real trained network deviates from this pointwise identity, the stated bounds need not govern its OOD behavior.","fun_headline_variants_meta":{"raw":{"variants":["OOD convergence bound depends on L2O feature size","Gradient-only L2O: theory-backed 10x speedup","Proved: OOD slowdown scales with input features in L2O","Robust L2O with guarantees: smaller features, faster OOD","New L2O method: tighter bounds, 10x faster convergence"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000273,"raw_usage":{"total_tokens":1682,"prompt_tokens":1037,"completion_tokens":645,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":653,"completion_tokens_details":{"reasoning_tokens":552}},"tokens_in":653,"tokens_out":645,"duration_ms":6881,"temperature":1.0,"reasoning_tokens":552,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T00:18:47.201660+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train a one-dimensional convex quadratic problem with an L2O model constructed so that Assumption 1 holds on a dense InD grid, then run the frozen model on an OOD objective $F'(x)=F(x+t)$ and check whether the per-iteration objective drop ever exceeds the right-hand side of Corollary 2; any violation would refute the bound. Alternatively, evaluate $N_1(z)$ and $N_2(z)$ on held-out InD points and find any point where they differ from $1/(2L)$ and $\\nabla f/(2L)$, which would show that the theorem's premise is not satisfied in practice.","supporting_citations":[{"cited_title":"Towards Constituting Mathematical Structures for Learning to Optimize","cited_arxiv_id":null,"evidence_quote":"Defines the Math-L2O update form and workflow that the paper builds on, and supplies the baseline L2O-PA used in experiments."},{"cited_title":"A fast iterative shrinkage- thresholding algorithm for linear inverse problems","cited_arxiv_id":null,"evidence_quote":"Defines FISTA, used both to generate optimal objective labels for the experimental evaluation and as a classical comparison baseline."},{"cited_title":"A method for stochastic optimization","cited_arxiv_id":null,"evidence_quote":"Adam is the optimizer used to train the proposed model and the learning-based baselines."},{"cited_title":"Ada-lista: Learned solvers adaptive to varying models","cited_arxiv_id":null,"evidence_quote":"Ada-LISTA is one of the learned-optimizer baselines in the experimental comparison."},{"cited_title":"Learning to learn by gradient descent by gradient descent","cited_arxiv_id":null,"evidence_quote":"L2O-DM is a black-box learning-to-optimize baseline included in the experimental comparison."},{"cited_title":"Learning gradient descent: Better generalization and longer horizons","cited_arxiv_id":null,"evidence_quote":"L2O-RNNprop is a baseline learned optimizer compared in the experiments."},{"cited_title":"Monotone operators and the proximal point algorithm","cited_arxiv_id":null,"evidence_quote":"The proximal point algorithm is the backbone used in the non-smooth and composite convergence analyses."}],"review_version":1}