{"id":"b53597d2-9643-4344-a033-3e5a789864c6","arxiv_id":"2412.00059","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"An LSTM-based model predicts coordinate-wise step sizes for BFGS and reports faster convergence while claiming theoretical guarantees that are only partially met.","lead":"This paper trains an LSTM network to pick separate step sizes for each coordinate during BFGS optimization, and reports up to 4x faster convergence than line search and hypergradient baselines. It also derives sufficient conditions for coordinate-wise step sizes, but the practical constraint used in the model does not actually guarantee those conditions.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The hard bound p∈(0,2) does not imply Theorem 2's p≤2γ/L; since γ≤L makes 2γ/L≤2, the claimed 'safe region' is weaker than the sufficient conditions, so the inherent theoretical guarantee is not established.","rationale":"The reader's weakest assumption correctly identifies the exact point. The paper has a coherent empirical story, but its own Appendix D acknowledges the simplification; the issue is not merely that γ or L are unknown, but that the fixed bound (0,2) is on the wrong side of the implication. This is internally inconsistent with the theorem, not merely outside consensus: a sufficient condition for stability is relaxed to a broader condition, with no argument that the broader condition remains sufficient. Since the abstract and introduction market the method as 'inherently respecting the derived theoretical guarantees', this is load-bearing. With a repair (e.g., estimating or bounding γ/L, or explicitly weakening the claim to an empirical one without theoretical guarantees) the paper could be conditionally accepted. I agree with the reader's CONDITIONAL verdict; no adjustment beyond it is needed.","tokens_in":14036,"tokens_out":6295,"duration_ms":58781,"concrete_test":"Analytic check: set f(x)=0.5(γ x_1^2 + L x_2^2) with γ=1, L=100, B_k=I, P_k=2I. The second error component evolves as e_{k+1,2}=(1−200)e_{k,2}, so ∥e_{k+1}∥>∥e_k∥ even though every p_i∈(0,2); this contradicts the claimed 'safe interval'. Model check: instrument BFGS-L2O on a least-squares test instance; at each iteration compute 2λ_min(B_k)/L (or a certified lower bound) and record the fraction of coordinates/iterations with p_{k,i} above this threshold. A nonzero fraction shows the implemented method operates outside the theorem's sufficient region, confirming that the theoretical guarantee is not in fact respected.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central theoretical claim is that the LSTM's sigmoid output P_k=diag(2σ(p_k)) 'inherently respects' Theorems 1–2. It does not. Theorem 2 requires 0<p_{k,i}≤2γ/L where γI⪯B_k. Since the remark itself notes γ≤L, we have 2γ/L≤2, so the interval (0,2) is strictly weaker than the theorem's upper bound; p=2 is admissible in the implementation but violates the sufficient condition whenever γ<L. The same issue affects Theorem 1, whose bound ∥P_k∥≤α/(L∥B_k^{-1}∥) can be far below 2. Section 4 and Appendix D describe the 0–2 constraint as a 'practical and robust simplification' and a 'safe operating region', but the simplification runs in the wrong direction: it enlarges the set of allowed step sizes rather than restricting it to the proven region. Thus the headline claim that the method achieves speedups 'while inherently respecting the derived theoretical guarantees' is unsupported by the stated theorems. The empirical speedups may still be real, but the paper's central stated justification for stability and convergence needs repair or explicit weakening.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a learn-to-optimize (L2O) method that uses an LSTM/MLP to generate coordinate-wise step-size matrices for BFGS updates. It first derives sufficient conditions for coordinate-wise step sizes in BFGS (Theorems 1–3) and then argues that the proposed architecture, which outputs step sizes in (0,2) via a scaled sigmoid, inherently respects these conditions. The method is evaluated on least squares, logistic regression, log-sum-exp problems, and a simple CNN training task, reporting up to 4x faster convergence in iterations compared to backtracking line search and hypergradient descent. The paper includes proofs in an appendix and a limitations section.","tokens_in":14293,"tokens_out":11837,"duration_ms":98965,"significance":"If the theoretical-to-implementation bridge were sound, the paper would be a useful contribution to the under-explored area of step-size tuning in quasi-Newton methods, combining a theoretical sufficient-condition analysis with a data-driven L2O policy. The empirical study is reasonably broad, covers several problem classes with increasing dimension, and reports consistent iteration-count speedups. The paper also provides complete proofs of the stated theorems and uses a standard meta-learning train/test split, which avoids circularity in the experimental protocol. However, the central claim that the implemented constraint (0,2) 'inherently respects' the derived sufficient conditions is not correct, and this affects the main narrative of guaranteed stability and convergence. The practical value of the empirical speedups is also limited by the absence of wall-clock time measurements and code. With a corrected theoretical justification or an explicitly weakened claim, the empirical contribution could still stand, but the current form overstates the theoretical grounding.","major_comments":[{"comment":"The implementation constraint P_k = diag(2σ(p_k)) with entries in (0,2) does not imply the sufficient condition 0 < p_{k,i} ≤ 2γ/L of Theorem 2. The remark that the upper bound 2γ/L 'simplifies to 2' is mathematically incorrect: since γ ≤ L, the ratio 2γ/L is at most 2, and it can be arbitrarily close to 0 when the Hessian is ill-conditioned. Therefore, allowing p_{k,i} up to 2 admits step sizes that violate Theorem 2 whenever γ < L. The same problem affects Theorem 1, whose upper bound ∥P_k∥_2 ≤ α/(L∥B_k^{-1}\\|_2) is strictly less than 2 because ∥B_k^{-1}\\|_2 ≥ 1/∥B_k\\|_2 and ∥B_k\\|_2 ≤ L, making the bound no larger than α < 2. Consequently, the abstract and Section 4 claim that the method 'inherently respects the derived theoretical guarantees' is unsupported. The paper should either enforce the actual problem-dependent bound (for instance, by estimating γ and L) or explicitly present the (0,2) constraint as a heuristic simplification that is motivated by, but not guaranteed to satisfy, the theorems.","section":"Section 3.3, Theorem 2 and its remark; Section 4"},{"comment":"The 'practical and robust simplification' described in Section 4 runs in the wrong direction: the theoretical safe region is a subset of (0,2), while the implementation permits all of (0,2), including values that can violate the sufficient conditions. Appendix D's limitation statement only notes that the constraint might preclude faster strategies outside the bounds; it does not acknowledge that the constraint can also admit unsafe step sizes. Because the claimed stability and convergence guarantees hinge on the step sizes satisfying the theorems, this omission must be corrected, either by changing the architecture or by explicitly stating that the guarantees do not apply to the implemented algorithm.","section":"Appendix D and Section 4"},{"comment":"All reported speedups are in iteration counts. The BFGS-L2O method incurs additional per-iteration cost from the LSTM and MLP forward passes, while the backtracking line search baseline also requires function evaluations. Without wall-clock time measurements, the headline claim of 'up to 4× faster convergence' is not established as an actual runtime speedup. Please include wall-clock comparisons or explicitly qualify the speedup as iteration-wise convergence.","section":"Section 5 (Experiments)"},{"comment":"The paper does not provide code, and the description of the L2O model is incomplete: the LSTM hidden size, MLP architecture, regularization coefficient λ, the number of inner-loop iterations K, and the exact training details are not specified beyond 200 Adam updates with batch size 64. Since the L2O model is the central methodological contribution, this level of detail is insufficient to reproduce or verify the empirical results. Releasing code and reporting the full hyperparameter settings would substantially strengthen the paper.","section":"Appendix B and code availability"}],"minor_comments":[{"comment":"The phrase 'In this chapter' should be 'In this section' for consistency with the rest of the paper.","section":"Section 2"},{"comment":"The formula P_k = α*_k I − (1/LR) v_k B^{-1}_k ∇f(x_k) appears dimensionally inconsistent: v_k is defined as a diagonal matrix, so v_k B^{-1}_k ∇f(x_k) is a vector, which cannot be subtracted from a matrix. Please clarify the intended construction, presumably involving diag(v_k ⊙ B^{-1}_k ∇f(x_k)) or a similar outer-product form.","section":"Section 3.1 and Appendix C"},{"comment":"The sentence 'Following the setup in [16] (from your main text)' contains a leftover template instruction and should be removed.","section":"Appendix B.3"},{"comment":"The second inequality in the proof uses a condition on λ_max(P_k), but the theorem statement gives a condition on ∥P^{-1}\\|_2, which is equivalent to a lower bound on λ_min(P_k). Please reconcile this mismatch.","section":"Appendix A.1, proof of Theorem 1"},{"comment":"Unlike Figure 4, Figure 1 shows a single curve without variance or error bars; adding multiple seeds would improve the reliability of the reported improvement.","section":"Figure 1"},{"comment":"In the proof of Theorem 2, T_k is not symmetric, so statements about 'the eigenvalue of T_k' should be phrased in terms of spectral radius or singular values rather than eigenvalues without qualification.","section":"Theorem 2 proof"}],"recommendation":"major_revision","confidential_remarks":"The central issue—the (0,2) bound not implying the sufficient conditions—is a load-bearing flaw in the paper's main claim. It is, however, fixable by either implementing an adaptive bound based on estimates of γ and L or by reframing the paper as a purely empirical L2O study with a heuristic motivation from the theory. The latter would substantially reduce the novelty, so the authors should be pushed to clarify whether the theoretical guarantees apply to the actual algorithm. The empirical results may still be publishable if the claims are correctly qualified and wall-clock time is provided. I would also suggest the editor ask for code release, as the paper's empirical contribution is not reproducible from the current description."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Three things to know. First, the paper is a genuine attempt: an LSTM that predicts coordinate-wise step sizes inside BFGS, with a regularization term pulling the step matrix toward the identity, motivated by a superlinear convergence theorem. That specific combination is new, and the empirical results, if reproducible, would be genuinely useful for the optimization subfield. Second, the theoretical scaffolding is not as solid as the abstract claims. The theorems give sufficient conditions that depend on L, γ, and B_k, but the implementation just clamps p to (0,2) via a sigmoid. That is a relaxation, not a simplification, of the theorem's bound p ≤ 2γ/L. The remark that “γ ≤ L so 2γ/L simplifies to 2” is wrong: it only gives an upper bound on the theorem's upper bound, so the implemented region is strictly larger than the proven region. As a result, the headline claim that the method “inherently respects the derived theoretical guarantees” does not hold. The authors acknowledge the simplification in the limitations, but they still call (0,2) a “safe operating region,” which is exactly where the logic fails. Third, the empirical section reports consistent iteration speedups (up to 4x) on least squares, logistic regression, log-sum-exp, and a small CNN. But there is no code, no wall-clock times, and the missing comparison to [15] (multidimensional backtracking) weakens the positioning. The CNN experiment reports training loss only, not test accuracy.\n\nWhat is genuinely good: the per-coordinate LSTM with weight sharing is a sensible design; the frequent inner-loop updates make sense because the step size shapes the Hessian approximation; the regularization term aligns with Theorem 3 and is a nice touch. The theorems, while variants of standard descent arguments, are cleanly stated and proved in the appendix. The paper is clear and readable.\n\nMy bottom line: the central theoretical claim needs repair before publication. Either estimate γ and L (or a proxy) and enforce the actual bound, or explicitly present (0,2) as a heuristic and drop the phrase “inherently respecting.” The empirical contribution can stand on its own if the code and timing data are released. I would send it to a serious referee, but with the expectation of major revision. Readers interested in L2O and quasi-Newton step-size control will get value from the empirical findings even if the theory needs rework.","headline":"Plausible L2O method with a broken theory-to-implementation bridge: the (0,2) step-size clamp is a relaxation of the theorem's bound, not a corollary, so the central guarantee claim needs rework.","tokens_in":788,"tokens_out":2094,"would_cite":false,"duration_ms":39311,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["65K10","90C53","68T07"],"pacs":[],"model":"deepseek-v4-flash","headline":"A learned LSTM supplies coordinate-wise step sizes for BFGS, up to 4× faster than line search and hypergradient descent, while respecting sufficient convergence conditions.","keywords":["coordinate-wise step sizes","BFGS","quasi-Newton methods","learn-to-optimize","LSTM step size prediction","superlinear convergence","hypergradient descent"],"falsifier":"Take a strongly convex quadratic whose Hessian has a very small eigenvalue $\\gamma$ relative to its largest eigenvalue $L$. Run the paper's BFGS-L2O method from a start point far from the optimum and record $\\|x_{k+1}-x^*\\|$; a single iteration where $\\max_i p_{k,i} > 2\\gamma/L$ and the distance to the optimum increases would violate Theorem 2 and falsify the claim that clamping to $(0,2)$ respects the derived conditions.","tokens_in":13780,"feed_emoji":"⚡","tokens_out":10633,"duration_ms":84486,"temperature":0.7,"pith_summary":"Step size tuning in BFGS is usually scalar, but this paper argues the extra freedom of a per-coordinate step size pays off even in second-order methods, where line search and hypergradient descent have limited success. It derives sufficient conditions on a diagonal step-size matrix $P_k$ under which BFGS still converges, keeps the distance to the optimum non-increasing, and retains superlinear convergence. It then trains a single coordinate-wise LSTM to output $P_k=\\mathrm{diag}(2\\sigma(p_k))$, bounding every step size to $(0,2)$ as a practical stand-in for those conditions. On least squares, logistic regression, log-sum-exp, and a small CNN task, the learned step sizes converge up to 4× faster than backtracking line search and hypergradient descent, with the advantage growing with dimension.","feed_headline":"Learned per-coordinate step sizes speed BFGS up to 4x","feed_subtitle":"A neural network sets a separate step for each variable, beating line search and hypergradient descent baselines.","key_machinery":"The load-bearing object is the diagonal coordinate-wise step-size matrix $P_k=\\mathrm{diag}(2\\sigma(p_k))$, produced by an LSTM plus an MLP and clamped so each entry lies in $(0,2)$. The theoretical support is a safe operating region defined by three sufficient conditions: the norm conditions of Theorem 1, the entrywise bound $0<p_{k,i}\\le 2\\gamma/L$ where $\\gamma$ is the smallest eigenvalue of the approximate Hessian $B_k$, and the asymptotic requirement $P_k\\to I$ from Theorem 3. The training objective $f(x_{k+1})+\\lambda\\|P_k-I\\|_F^2$ operationalizes these conditions: the first term drives fast descent, while the regularization encourages the identity limit that preserves BFGS's superlinear convergence.","core_discovery":"The paper's central claim is that coordinate-wise step sizes for BFGS can be generated by a neural network and still respect the convergence guarantees of quasi-Newton methods. Theoretically, it proves three sufficient conditions: a norm bound on $P_k$ and a related bound on $P_k^{-1}$ force the gradient norms to vanish (Theorem 1); for convex $L$-smooth objectives with uniformly lower-bounded Hessian approximations, entrywise bounds $0<p_{k,i}\\le 2\\gamma/L$ make each step monotonically closer to the minimizer (Theorem 2); and if $P_k\\to I$ along a convergent trajectory, BFGS keeps its superlinear rate (Theorem 3). On the algorithmic side, an LSTM shared across coordinates maps $(x_k,\\nabla f(x_k),B_k^{-1}\\nabla f(x_k))$ to a diagonal $P_k=\\mathrm{diag}(2\\sigma(p_k))$, trained to minimize the next objective value plus a regularization $\\lambda\\|P_k-I\\|_F^2$ that pushes the step sizes toward identity near the optimum. The paper reports that this trained policy converges up to 4× faster than backtracking line search and hypergradient descent across the tested problems.","pith_inferences":["The practical simplification that clamps step sizes to $(0,2)$ is only guaranteed by Theorem 2 when $2\\gamma/L\\ge 2$; on ill-conditioned objectives with $\\gamma\\ll L$, entries near 2 could fall outside the proven safe region, so estimating $\\gamma$ locally and shrinking the upper bound would make the guarantee airtight.","Because the LSTM is trained on a distribution of random problem instances, its speedup is likely tied to shared structure in that distribution; on objectives whose geometry differs strongly from the training set, the learned policy may degrade to or below the line-search baseline.","The frequent outer-loop updates after every inner iteration are a deliberate response to the dual role of step sizes in BFGS, since a step affects both the iterate and the Hessian approximation; testing longer unrolls or a surrogate that predicts curvature quality could reveal the trade-off between responsiveness and meta-training stability."],"forward_implications":["On the tested least squares and log-sum-exp problems, BFGS with learned coordinate-wise step sizes reaches the target objective in roughly 25% to 75% fewer iterations than line search; for log-sum-exp the speedup grows from 2× at dimension 100 to 4× at dimension 500.","The same shared LSTM handles different problem dimensions without adding parameters, so the method scales to higher-dimensional optimization tasks.","Near the optimum the regularization term pushes $P_k$ toward the identity, so the learned step sizes hand control back to the unmodified BFGS update and preserve its superlinear convergence.","Hypergradient descent on the step-size matrix gives only marginal gains after the first few BFGS iterations, and can be unstable on neural network training, whereas the learned policy is reported to be more stable."],"supporting_citations":[{"why":"Provides the BFGS update, secant equation, and the convergence-rate framework that Theorems 1 and 3 extend.","marker":"[25]"},{"why":"Supplies the characterization of superlinear convergence used in the proof of Theorem 3.","marker":"[10]"},{"why":"Supplies the LSTM-based step-size prediction architecture, the unrolling and training conventions, and the sparse least-squares dataset recipe the method adopts.","marker":"[16]"},{"why":"Establishes the learning-to-optimize paradigm and the coordinate-wise shared-network trick used for dimensionality-independent scaling.","marker":"[2]"},{"why":"Is the multidimensional backtracking approach for per-coordinate step sizes in first-order methods that the paper argues does not transfer cleanly to second-order settings.","marker":"[15]"},{"why":"Introduces hypergradient-based step-size tuning, the baseline the paper shows gives only marginal gains inside BFGS.","marker":"[18]"}],"fun_headline_variants":["LSTM tunes each BFGS step coordinate-wise, up to 4x faster","Neural network picks step sizes for BFGS: 4x speedup","Learned per-variable step sizes give BFGS a 4x boost","Deep learning sets BFGS step sizes, achieving 4x acceleration"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that bounding every coordinate-wise step size to the interval $(0,2)$ automatically satisfies the paper's sufficient conditions; that reduction identifies $2\\gamma/L$ with $2$, which is safe only when the smallest curvature $\\gamma$ of the Hessian approximation is comparable to the smoothness constant $L$, not when $\\gamma\\ll L$.","fun_headline_variants_meta":{"raw":{"variants":["LSTM tunes each BFGS step coordinate-wise, up to 4x faster","Neural network picks step sizes for BFGS: 4x speedup","Learned per-variable step sizes give BFGS a 4x boost","Deep learning sets BFGS step sizes, achieving 4x acceleration"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000432,"raw_usage":{"total_tokens":2224,"prompt_tokens":989,"completion_tokens":1235,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":605,"completion_tokens_details":{"reasoning_tokens":1153}},"tokens_in":605,"tokens_out":1235,"duration_ms":11366,"temperature":1.0,"reasoning_tokens":1153,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T13:29:31.212054+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a strongly convex quadratic whose Hessian has a very small eigenvalue $\\gamma$ relative to its largest eigenvalue $L$. Run the paper's BFGS-L2O method from a start point far from the optimum and record $\\|x_{k+1}-x^*\\|$; a single iteration where $\\max_i p_{k,i} > 2\\gamma/L$ and the distance to the optimum increases would violate Theorem 2 and falsify the claim that clamping to $(0,2)$ respects the derived conditions.","supporting_citations":[{"cited_title":"safe operating region","cited_arxiv_id":null,"evidence_quote":"Provides the BFGS update, secant equation, and the convergence-rate framework that Theorems 1 and 3 extend."},{"cited_title":"A Characterization of Superlinear Convergence and Its Application to Quasi-Newton Methods","cited_arxiv_id":null,"evidence_quote":"Supplies the characterization of superlinear convergence used in the proof of Theorem 3."},{"cited_title":"Towards Constituting Mathe- matical Structures for Learning to Optimize","cited_arxiv_id":null,"evidence_quote":"Supplies the LSTM-based step-size prediction architecture, the unrolling and training conventions, and the sparse least-squares dataset recipe the method adopts."},{"cited_title":"Learning to Learn by Gradient Descent by Gradient Descent","cited_arxiv_id":null,"evidence_quote":"Establishes the learning-to-optimize paradigm and the coordinate-wise shared-network trick used for dimensionality-independent scaling."},{"cited_title":"Searching for Optimal Per-Coordinate Step-Sizes with Multidimensional Backtracking","cited_arxiv_id":null,"evidence_quote":"Is the multidimensional backtracking approach for per-coordinate step sizes in first-order methods that the paper argues does not transfer cleanly to second-order settings."},{"cited_title":"Gradient-Based Hyperparameter Optimization through Reversible Learning","cited_arxiv_id":null,"evidence_quote":"Introduces hypergradient-based step-size tuning, the baseline the paper shows gives only marginal gains inside BFGS."}],"review_version":1}