{"id":"ce62593c-02f5-446e-be2b-00572eeab170","arxiv_id":"2411.15717","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A machine-learning framework that learns a shared hyperparameter sequence for first-order optimization solvers, achieving order-of-magnitude speedups with only 10 training instances.","lead":"This paper presents a framework that learns step sizes and other hyperparameters of optimization algorithms from just a handful of example problems, so the same solver runs faster on new, similar problems. If it works broadly, it could make real-time control, signal processing, and machine learning tasks much cheaper to solve.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 'guaranteed convergence' claim for the logistic regression example is not established for unseen instances: a single steady-state step size cannot satisfy 0<θ_H<2/L(x) if L(x) varies, and the safeguard trigger is calibrated to training-average optimal values rather than per-instance f*(x).","rationale":"The reader's weakest assumption focused on fixed matrices P and A, which is a real scope limitation but is explicitly stated in Section 4 for OSQP/SCS and does not by itself break the convergence guarantee. The more load-bearing issue is that the convergence guarantee for gradient descent depends on a data-dependent smoothness constant L(x), and the paper's logistic regression example uses a single shared θ_H across instances with varying L(x). No uniform Lipschitz bound is reported, and the safeguarding mechanism that might backstop convergence uses a training-average optimal value f* rather than the test instance's own f*(x), making its trigger condition miscalibrated for unseen data. This directly threatens the central claim of guaranteed convergence on unseen instances, which is a stronger concern than scope. The rest of the paper appears sound: the closed-form two/three-step analyses are carefully derived, the generalization bounds are honestly post-training, and the code is public. The concern is concrete and testable, so a conditional verdict remains appropriate; the authors should report per-instance L(x) coverage and safeguard trigger statistics before the guarantee can be accepted as stated.","tokens_in":36841,"tokens_out":13325,"duration_ms":126734,"concrete_test":"Re-run the logistic regression experiment (Section 7.1.2) with instrumentation: for each of the 1000 test problems, compute a valid upper bound on the true smoothness constant L(x) (e.g., the largest eigenvalue of the Hessian at the optimum) and record whether the learned steady-state step size θ_H satisfies 0<θ_H<2/L(x). Also re-implement the safeguard using the per-instance optimal value f*(x) rather than the training average, and count how many test problems would trigger the safeguard under this correct calibration. If θ_H violates the bound on a non-negligible fraction of test instances and the safeguard does not trigger (or triggers only under the mis-calibrated training-average f*), then the convergence guarantee for unseen data is not delivered.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's headline guarantee ('guaranteed to converge to an optimal solution') requires that the steady-state hyperparameters satisfy the convergence condition for every test instance. In Table 1, gradient descent requires 0<θ_H<2/L, where L is the smoothness constant of f(z,x). In the logistic regression experiment (Section 7.1.2), the parameter x is the full dataset, so L(x) varies considerably across instances; a single learned θ_H cannot be assumed to satisfy the bound for all unseen instances unless the authors explicitly enforce it against a uniform upper bound on L. The paper does not report such a bound, nor how θ_H was chosen in that example. The only mechanism that could rescue convergence on instances where the bound is violated is the safeguarding fallback (Section 5.5), but its trigger condition in Appendix B is stated as a comparison involving f* defined as 'the average optimal value over the training instances'. For a test instance whose optimal value differs from that average, the estimated suboptimality is biased, so the safeguard can either fail to trigger during divergence or fire on healthy iterates. The asymptotic convergence guarantee for unseen data is therefore not actually delivered by the analysis as written; it is an empirical hope unless the Lipschitz/safeguard calibration issue is resolved.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes LAH, a framework for learning a shared hyperparameter sequence for parametric fixed-point and convex optimization solvers. The architecture consists of a step-varying phase of H iterations followed by a steady-state phase whose hyperparameters are chosen so that classical convergence theory applies. Training minimizes the mean squared error to ground-truth solutions using progressive B-step lookahead, with closed-form one-, two-, and three-step solutions for gradient descent and unconstrained quadratic minimization, and backpropagation for other algorithms. The paper also constructs validation-set generalization bounds and quantile bounds, and reports experiments on ridge and logistic regression, lasso, image deblurring, robust Kalman filtering, and max cut, using only 10 training instances in most examples.","tokens_in":37156,"tokens_out":10554,"duration_ms":102415,"significance":"If the claims hold, the paper makes a useful contribution: it shows that a very small number of shared hyperparameters can be learned with little data while retaining an asymptotic convergence guarantee, and it provides closed-form lookahead solutions plus numerically meaningful generalization bounds. The full proofs in the appendix and the public code repository are strengths, as is the breadth of the numerical study across GD, proximal GD, OSQP, and SCS. The main weaknesses are that the headline convergence guarantee is not actually established for the logistic regression experiment, the safeguarding mechanism is underspecified and biased as written, and the empirical tables report no measure of dispersion.","major_comments":[{"comment":"The asymptotic convergence guarantee claimed in the abstract is not established for the logistic regression experiment. Table 1 states that the steady-state gradient-descent step size must satisfy 0 < theta_H < 2/L, but in Section 7.1.2 the parameter x is the full dataset (v_j, y_j), so the smoothness constant L = L(x) varies across instances. Since LAH learns a single theta_H shared by all instances, the condition 0 < theta_H < 2/L cannot be verified for unseen x unless the paper gives a uniform upper bound L_max over the parameter distribution and explicitly enforces theta_H < 2/L_max. No such bound or enforcement is reported. The safeguarding mechanism in Appendix B does not fill this gap because its trigger uses f*, the average optimal value over the training instances, which is not a valid per-instance optimality certificate for test instances. Please provide either a rigorous per-instance convergence argument, a repaired fallback with per-instance certificates, or a restriction of the 'guaranteed to converge' claim to the settings where the bound is actually enforced.","section":"Section 7.1.2, Table 1, and Appendix B"},{"comment":"The safeguarding mechanism is load-bearing for the logistic and lasso experiments, but as written it is not fully specified. The trigger condition in Appendix B, 'f(z^k(x), x) - f* > 10(f(z^k(x), x) - f*)', is self-referential and presumably a typo for comparing consecutive iterates. If the intended condition is f(z^k(x), x) - f* > 10(f(z^{k-1}(x), x) - f*), then replacing the per-instance optimal value with the training-average f* biases the test-time criterion: for instances with atypical optimal values, the safeguard can either fail to trigger during divergence or trigger on healthy iterates. The paper should state the corrected condition, justify the factor of 10, and report how often the safeguard triggers in every experiment where it is used; currently only the lasso example gives a trigger count.","section":"Section 5.5 and Appendix B"},{"comment":"The empirical claims are reported as a single mean iteration count per tolerance with no measure of dispersion. Given that the training set has only 10 instances and both training and test instances are randomly generated, the reported speedups in Tables 3-8 could be sensitive to the particular random draw. Please report standard deviations or quantiles over multiple independent training seeds, or at least the raw per-instance distributions, for LAH and the main baselines. This is needed to support the data-efficiency claim rather than a claim about one favorable draw.","section":"Section 7, Tables 3-8"},{"comment":"Theorem 6 as stated contains a mathematical typo in the central formula for the stochastic lookahead problem. It defines \\bar z^k_j = (a^k_j)^T mu + (a^k_j)^T Sigma a^k_j, and the proof writes E[(a_j^T x)^2] = mu^T a_j + a_j^T Sigma a_j. For x ~ N(mu, Sigma), the correct identity is E[(a_j^T x)^2] = (a_j^T mu)^2 + a_j^T Sigma a_j, so the displayed formula is missing a square on the first term and is not a valid second moment. Because this formula underlies the claim that no training instances are needed for the ridge regression example, please correct it and re-verify the corresponding numerical results.","section":"Section 5.3, Theorem 6"}],"minor_comments":[{"comment":"The statement that 'we only use 10 problem instances to train the hyperparameters in all of our examples' is not literal, since the ridge regression example uses zero training instances and solves the stochastic problem directly; please rephrase to indicate that at most 10 instances are used.","section":"Abstract and Section 7.1.1"},{"comment":"There are several typos: 'identiy matrix' in the notation paragraph, 'Kullback-Liebler' should be 'Kullback-Leibler', 'we firsting generating H different trajectories' in Section 7.4.1, and 'outpeforms' in Section 7.3.1. The cross-reference to 'Problem (7.1.1)' in Section 7.1.1 is not meaningful because that problem is not numbered.","section":"Notation and typos"},{"comment":"The fixed P and A assumption is stated only in Section 4; since it is central to the factorization-caching efficiency claim and to the transfer of hyperparameters across instances, it should be stated as a scope limitation in the introduction or abstract.","section":"Section 4"},{"comment":"The smoothness parameter L is used to define the comparison algorithms and the steady-state constraint, but L is not defined for the logistic regression problem; please state explicitly how L is computed for each instance or how a single value is used.","section":"Section 7.1.2"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope for math.OC and the core idea is attractive. The main issues are fixable: the logistic convergence guarantee needs an explicit uniform bound or a softened claim, the safeguarding condition needs correction and per-instance justification, the stochastic lookahead formula has a typographical error, and the empirical reporting needs variance measures. I do not see a novelty or attribution concern; the reliance on the authors' own prior work is clearly cited."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things you should know. First, this is a real contribution, not a repackaging: shared hyperparameter sequences, a two-phase architecture with a provably convergent steady state, closed-form two- and three-step lookahead for quadratic minimization, and KL-based generalization bounds are combined in a way I have not seen. The math in Theorems 1–7 looks correct, the proofs are detailed, and the numerical scope is broad—gradient descent, proximal gradient descent, OSQP, SCS, with code released. The data efficiency claim also holds up: ten training instances is genuinely surprising.\n\nSecond, the abstract's guarantee that the learned optimizer is 'guaranteed to converge to an optimal solution' is not actually established for all the experiments as written. The steady-state phase only ensures convergence if the fixed hyperparameters satisfy the classical range condition, e.g., 0 < theta_H < 2/L for gradient descent. In ridge regression, lasso, image deblurring, Kalman filtering, and maxcut, the problem data that determines L (or the analog for ADMM) is indeed fixed across instances, so the guarantee is legitimate there. But in logistic regression, the parameter x is the whole dataset and L(x) varies across instances. A single learned theta_H cannot satisfy 0 < theta_H < 2/L(x) for every unseen x unless the authors enforce it against a uniform upper bound on L over the instance distribution. They do not report such a bound or say how theta_H was chosen in that example.\n\nThe safeguarding mechanism is supposed to rescue this, but it has its own issue. Appendix B states the trigger uses f* defined as the average optimal value over the training instances. For a test instance whose optimal value differs from that average, the estimated suboptimality is biased, so the safeguard can either fail to trigger during divergence or fire on healthy iterates. The stress-test note lands here. This is the paper's softest spot, and it is a load-bearing one for the logistic regression result. The fix is straightforward: enforce a certified uniform bound on L (or a per-instance safeguard that uses an unbiased estimate of f*(x)), and report safeguard trigger rates across the test set.\n\nMinor issues: Tables 3–8 report mean iterations without standard deviations, and the generalization bounds for logistic regression are wide—the strong test performance is not fully certified. Also, the fixed-matrix assumption stated in Section 4 is clearly acknowledged but limits the scope; the factorization caching claim depends on it.\n\nOverall, the central framework is sound and the flaws are overclaiming and missing calibration details, not fundamental unsoundness. The paper deserves a serious referee. I would send it to review and ask for a major revision that tightens the convergence claim, fixes the safeguard calibration, and adds error bars and trigger statistics.","headline":"A genuinely useful learning-to-optimize framework with sound closed-form results; the headline convergence guarantee for varying-instance problems like logistic regression rests on an unverified uniform Lipschitz bound and a biased safeguard.","tokens_in":37643,"tokens_out":2406,"would_cite":true,"duration_ms":25773,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["90C25","90C06","68T05"],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that learned per-iteration hyperparameter sequences can accelerate first-order solvers on parametric convex optimization problems, using only 10 training instances while preserving guaranteed convergence.","keywords":["learning to optimize","parametric convex optimization","hyperparameter learning","first-order methods","fixed-point iterations","operator splitting","generalization guarantees","ADMM"],"falsifier":"Run LAH on a fixed-matrix parametric family with 10 training instances and measure the 97.5th percentile upper bound on the residual after a fixed step budget; if the bound is vacuous or the learned schedule is slower than vanilla gradient descent with step size $1/L$, the claim of data-efficient acceleration is falsified.","tokens_in":36640,"feed_emoji":"⚡","tokens_out":6662,"duration_ms":54789,"temperature":0.7,"pith_summary":"The paper claims that the hyperparameter sequence of a first-order optimization algorithm—such as step sizes in gradient descent—can be learned from data to solve parametric convex problems much faster than hand-tuned schedules. The framework, called LAH, runs fixed-point iterations in which the same hyperparameters are shared across all problem instances, with a step-varying phase followed by a steady-state phase that guarantees eventual convergence to an optimal solution. Training minimizes the mean squared error to a ground-truth solution, and in several cases the training subproblems admit closed-form solutions. The paper reports that only 10 training instances are needed across all experiments, and that the learned schedules outperform neural-network warm starts and learned metrics on control, signal processing, and machine learning problems while carrying high-probability performance guarantees on unseen data.","feed_headline":"Ten training instances learn fast solver schedules that beat tuning","feed_subtitle":"A two-phase step-size schedule preserves convergence guarantees while cutting iterations on control, imaging, and ML problems.","key_machinery":"The central object is the two-phase fixed-point iteration $z_{k+1} = T_{\\theta_k}(z_k, x)$ for $k < H$ and $z_{k+1} = T_{\\theta_H}(z_k, x)$ for $k \\geq H$, where $T$ is the fixed-point map of the algorithm (gradient descent, proximal gradient descent, OSQP, or SCS) and $\\theta_k$ are the learned hyperparameters shared across all problem instances. The step-varying phase is what purchases speed; the steady-state phase is what purchases convergence, since standard theory applies once the hyperparameters are constant and within the usual ranges. The training machinery is progressive $B$-step lookahead: at each stage only $B$ hyperparameters are optimized, and for gradient descent the one-step problem reduces to a least squares problem while the two- and three-step quadratic problems reduce to finding roots of a quadratic and a cubic. The generalization machinery is the KL bound in Theorem 8, inverted over a grid of tolerances with a union bound to give quantile bounds on the performance metric.","core_discovery":"The central claim is that a two-phase hyperparameter schedule shared across parametric instances makes a fixed-point solver both fast and safe: the step-varying phase is free to use aggressive, iteration-specific hyperparameters, while the steady-state phase returns to a provably convergent constant hyperparameter, so the whole trajectory converges to an optimal solution no matter what the learned phase did. Because the hyperparameters are shared, the expensive matrix factorizations in solvers such as OSQP and SCS can be computed offline once and reused for every instance. The paper further claims that the one-step gradient-descent training problem is a least squares problem, that the two- and three-step problems for unconstrained quadratic minimization have closed-form global solutions, and that when the parameter distribution is Gaussian the training problem can be solved without any sampled instances. Finally, it claims that KL-based sample convergence bounds, computed on a validation set of 1000 instances, provide non-vacuous upper and lower quantile guarantees for the learned optimizer's performance on unseen data.","pith_inferences":["The fixed-matrix assumption is the most natural boundary of the method: if $P$ and $A$ vary across instances, the shared hyperparameter sequence will not transfer and the offline factorization caching ceases to apply, so extending LAH would require learning a mapping from matrix features to schedules.","The closed-form results for quadratic minimization suggest a general principle: the step-size schedule that minimizes a finite-horizon objective over a distribution is available in closed form whenever the residual covariance can be tracked, which may extend to other linear fixed-point iterations such as Anderson acceleration.","The data efficiency of using 10 instances indicates that the effective sample complexity is tied to the number of hyperparameters being learned (dozens of scalars), not the problem dimension, so the method should scale to larger problems without needing more training data.","One could test whether the learned schedules transfer across distributions with the same fixed matrices but different parameter distributions; the paper's convergence guarantees hold for any parameter, but the speed gains might degrade if the test distribution shifts far from the training distribution."],"forward_implications":["First-order solvers on parametric convex problems can be substantially accelerated using only 10 training instances, turning the solve-time bottleneck into a one-time offline training cost.","For parametric OSQP and SCS, all matrix factorizations can be done offline, so each new instance only requires cheap linear-system solves with cached factorizations.","Because the steady-state phase uses fixed hyperparameters, the learned optimizer can be evaluated on any number of iterations and inherits the underlying solver's convergence guarantee.","The learned schedules routinely use step sizes far outside the classical convergence range during the step-varying phase, suggesting that large-step strategies can be effective when balanced by a safe steady-state phase.","High-probability upper and lower quantile bounds on performance can be computed for unseen data, using only 1000 validation instances."],"supporting_citations":[{"why":"Supplies the KL sample convergence bound used in Theorem 8 to derive high-probability upper and lower bounds on risk for unseen data.","marker":"[49]"},{"why":"Provides Young's Chebyshev step sizes and the worst-case rate for quadratic minimization that Theorem 7 compares the learned schedule against.","marker":"[69]"},{"why":"The OSQP operator-splitting solver whose step-size and penalty hyperparameters LAH learns and whose linear-system structure enables factorization caching.","marker":"[65]"},{"why":"The SCS splitting conic solver whose scaling and step hyperparameters LAH learns for the Kalman filtering and maxcut examples.","marker":"[54]"},{"why":"Provides the learned-warm-start baseline (L2WS) and the numerical setups for the image deblurring and robust Kalman filtering experiments.","marker":"[61]"},{"why":"The learned-metric baseline (LM) that predicts hyperparameters per instance, which LAH is compared against and which motivates the shared-hyperparameter design.","marker":"[46]"},{"why":"Supplies the progressive training and learning-to-optimize framework that LAH's B-step lookahead training procedure builds on.","marker":"[22]"}],"fun_headline_variants":["Two-phase learned step sizes: fast and provably convergent","Learn solver step schedules from just 10 parameter instances","Learned step schedules keep solvers fast and safely convergent","10 instances train two-phase schedule for fast, provable convergence"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The problem-defining matrices $P$ and $A$ must be the same for all parametric instances, so that one shared hyperparameter sequence transfers across instances and the cached matrix factorizations remain valid.","fun_headline_variants_meta":{"raw":{"variants":["Two-phase learned step sizes: fast and provably convergent","Learn solver step schedules from just 10 parameter instances","Learned step schedules keep solvers fast and safely convergent","10 instances train two-phase schedule for fast, provable convergence"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001121,"raw_usage":{"total_tokens":4703,"prompt_tokens":1026,"completion_tokens":3677,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":642,"completion_tokens_details":{"reasoning_tokens":3610}},"tokens_in":642,"tokens_out":3677,"duration_ms":26315,"temperature":1.0,"reasoning_tokens":3610,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T14:00:51.627897+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run LAH on a fixed-matrix parametric family with 10 training instances and measure the 97.5th percentile upper bound on the residual after a fixed step budget; if the bound is vacuous or the learned schedule is slower than vanilla gradient descent with step size $1/L$, the claim of data-efficient acceleration is falsified.","supporting_citations":[{"cited_title":"Langford and R","cited_arxiv_id":null,"evidence_quote":"Supplies the KL sample convergence bound used in Theorem 8 to derive high-probability upper and lower bounds on risk for unseen data."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides Young's Chebyshev step sizes and the worst-case rate for quadratic minimization that Theorem 7 compares the learned schedule against."},{"cited_title":"Stellato, G","cited_arxiv_id":null,"evidence_quote":"The OSQP operator-splitting solver whose step-size and penalty hyperparameters LAH learns and whose linear-system structure enables factorization caching."},{"cited_title":"O’Donoghue, Operator splitting for a homogeneous embedding of the linear complemen- tarity problem, SIAM Journal on Optimization, 31 (2021), pp","cited_arxiv_id":null,"evidence_quote":"The SCS splitting conic solver whose scaling and step hyperparameters LAH learns for the Kalman filtering and maxcut examples."},{"cited_title":"Sambharya, G","cited_arxiv_id":null,"evidence_quote":"Provides the learned-warm-start baseline (L2WS) and the numerical setups for the image deblurring and robust Kalman filtering experiments."},{"cited_title":"Metric Learning to Accelerate Convergence of Operator Splitting Methods for Differentiable Parametric Programming","cited_arxiv_id":"2404.00882","evidence_quote":"The learned-metric baseline (LM) that predicts hyperparameters per instance, which LAH is compared against and which motivates the shared-hyperparameter design."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the progressive training and learning-to-optimize framework that LAH's B-step lookahead training procedure builds on."}],"review_version":1}